Result | FAILURE |
Tests | 23 failed / 774 succeeded |
Started | |
Elapsed | 47m57s |
Revision | master |
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(OnRootMismatch\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\snew\spod\sfsgroup\sapplied\sto\svolume\scontents$'
test/e2e/storage/testsuites/fsgroupchangepolicy.go:216 Jan 28 17:48:18.961: Unexpected error: <*errors.errorString | 0xc001bce8d0>: { s: "pod \"pod-270e9e95-df4d-4169-9c6c-9329994f97a6\" is not Running: timed out waiting for the condition", } pod "pod-270e9e95-df4d-4169-9c6c-9329994f97a6" is not Running: timed out waiting for the condition occurred test/e2e/storage/testsuites/fsgroupchangepolicy.go:264from junit_03.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy test/e2e/framework/framework.go:187 �[1mSTEP�[0m: Creating a kubernetes client Jan 28 17:40:05.505: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename fsgroupchangepolicy �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents test/e2e/storage/testsuites/fsgroupchangepolicy.go:216 Jan 28 17:40:07.011: INFO: Creating resource for dynamic PV Jan 28 17:40:07.011: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass fsgroupchangepolicy-7281-e2e-scnx5nh �[1mSTEP�[0m: creating a claim Jan 28 17:40:07.229: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-7281 with fsgroup 1000 Jan 28 17:43:08.525: INFO: Pod fsgroupchangepolicy-7281/pod-4cc498ec-12fb-4bcf-a472-144d26af39ea started successfully �[1mSTEP�[0m: Creating a sub-directory and file, and verifying their ownership is 1000 Jan 28 17:43:08.525: INFO: ExecWithOptions {Command:[/bin/sh -c touch /mnt/volume1/file1] Namespace:fsgroupchangepolicy-7281 PodName:pod-4cc498ec-12fb-4bcf-a472-144d26af39ea ContainerName:write-pod Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false} Jan 28 17:43:08.525: INFO: >>> kubeConfig: /root/.kube/config Jan 28 17:43:08.526: INFO: ExecWithOptions: Clientset creation Jan 28 17:43:08.526: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io/api/v1/namespaces/fsgroupchangepolicy-7281/pods/pod-4cc498ec-12fb-4bcf-a472-144d26af39ea/exec?command=%2Fbin%2Fsh&command=-c&command=touch+%2Fmnt%2Fvolume1%2Ffile1&container=write-pod&container=write-pod&stderr=true&stdout=true) Jan 28 17:43:09.898: INFO: ExecWithOptions {Command:[/bin/sh -c ls -l /mnt/volume1/file1] Namespace:fsgroupchangepolicy-7281 PodName:pod-4cc498ec-12fb-4bcf-a472-144d26af39ea ContainerName:write-pod Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false} Jan 28 17:43:09.898: INFO: >>> kubeConfig: /root/.kube/config Jan 28 17:43:09.899: INFO: ExecWithOptions: Clientset creation Jan 28 17:43:09.899: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io/api/v1/namespaces/fsgroupchangepolicy-7281/pods/pod-4cc498ec-12fb-4bcf-a472-144d26af39ea/exec?command=%2Fbin%2Fsh&command=-c&command=ls+-l+%2Fmnt%2Fvolume1%2Ffile1&container=write-pod&container=write-pod&stderr=true&stdout=true) Jan 28 17:43:11.280: INFO: pod fsgroupchangepolicy-7281/pod-4cc498ec-12fb-4bcf-a472-144d26af39ea exec for cmd ls -l /mnt/volume1/file1, stdout: -rw-r--r-- 1 root 1000 0 Jan 28 17:43 /mnt/volume1/file1, stderr: Jan 28 17:43:11.280: INFO: stdout split: [-rw-r--r-- 1 root 1000 0 Jan 28 17:43 /mnt/volume1/file1], expected gid: 1000 Jan 28 17:43:11.280: INFO: ExecWithOptions {Command:[/bin/sh -c mkdir /mnt/volume1/subdir] Namespace:fsgroupchangepolicy-7281 PodName:pod-4cc498ec-12fb-4bcf-a472-144d26af39ea ContainerName:write-pod Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false} Jan 28 17:43:11.280: INFO: >>> kubeConfig: /root/.kube/config Jan 28 17:43:11.281: INFO: ExecWithOptions: Clientset creation Jan 28 17:43:11.281: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io/api/v1/namespaces/fsgroupchangepolicy-7281/pods/pod-4cc498ec-12fb-4bcf-a472-144d26af39ea/exec?command=%2Fbin%2Fsh&command=-c&command=mkdir+%2Fmnt%2Fvolume1%2Fsubdir&container=write-pod&container=write-pod&stderr=true&stdout=true) Jan 28 17:43:12.663: INFO: ExecWithOptions {Command:[/bin/sh -c touch /mnt/volume1/subdir/file2] Namespace:fsgroupchangepolicy-7281 PodName:pod-4cc498ec-12fb-4bcf-a472-144d26af39ea ContainerName:write-pod Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false} Jan 28 17:43:12.663: INFO: >>> kubeConfig: /root/.kube/config Jan 28 17:43:12.664: INFO: ExecWithOptions: Clientset creation Jan 28 17:43:12.664: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io/api/v1/namespaces/fsgroupchangepolicy-7281/pods/pod-4cc498ec-12fb-4bcf-a472-144d26af39ea/exec?command=%2Fbin%2Fsh&command=-c&command=touch+%2Fmnt%2Fvolume1%2Fsubdir%2Ffile2&container=write-pod&container=write-pod&stderr=true&stdout=true) Jan 28 17:43:14.039: INFO: ExecWithOptions {Command:[/bin/sh -c ls -l /mnt/volume1/subdir/file2] Namespace:fsgroupchangepolicy-7281 PodName:pod-4cc498ec-12fb-4bcf-a472-144d26af39ea ContainerName:write-pod Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false} Jan 28 17:43:14.039: INFO: >>> kubeConfig: /root/.kube/config Jan 28 17:43:14.040: INFO: ExecWithOptions: Clientset creation Jan 28 17:43:14.040: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io/api/v1/namespaces/fsgroupchangepolicy-7281/pods/pod-4cc498ec-12fb-4bcf-a472-144d26af39ea/exec?command=%2Fbin%2Fsh&command=-c&command=ls+-l+%2Fmnt%2Fvolume1%2Fsubdir%2Ffile2&container=write-pod&container=write-pod&stderr=true&stdout=true) Jan 28 17:43:15.444: INFO: pod fsgroupchangepolicy-7281/pod-4cc498ec-12fb-4bcf-a472-144d26af39ea exec for cmd ls -l /mnt/volume1/subdir/file2, stdout: -rw-r--r-- 1 root 1000 0 Jan 28 17:43 /mnt/volume1/subdir/file2, stderr: Jan 28 17:43:15.444: INFO: stdout split: [-rw-r--r-- 1 root 1000 0 Jan 28 17:43 /mnt/volume1/subdir/file2], expected gid: 1000 �[1mSTEP�[0m: Deleting Pod fsgroupchangepolicy-7281/pod-4cc498ec-12fb-4bcf-a472-144d26af39ea Jan 28 17:43:15.444: INFO: Deleting pod "pod-4cc498ec-12fb-4bcf-a472-144d26af39ea" in namespace "fsgroupchangepolicy-7281" Jan 28 17:43:15.661: INFO: Wait up to 5m0s for pod "pod-4cc498ec-12fb-4bcf-a472-144d26af39ea" to be fully deleted �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-7281 with fsgroup 2000 Jan 28 17:48:18.961: FAIL: Unexpected error: <*errors.errorString | 0xc001bce8d0>: { s: "pod \"pod-270e9e95-df4d-4169-9c6c-9329994f97a6\" is not Running: timed out waiting for the condition", } pod "pod-270e9e95-df4d-4169-9c6c-9329994f97a6" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc002cbfb00, 0xc0026c5060, 0x0, {0xc001274538, 0x4}, {0xc00127453c, 0x4}) test/e2e/storage/testsuites/fsgroupchangepolicy.go:264 +0x151 k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() test/e2e/storage/testsuites/fsgroupchangepolicy.go:253 +0x556 k8s.io/kubernetes/test/e2e.RunE2ETests(0x25761d7?) test/e2e/e2e.go:130 +0x6bb k8s.io/kubernetes/test/e2e.TestE2E(0x24e52d9?) test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc00075d040, 0x741f9a8) /usr/local/go/src/testing/testing.go:1446 +0x10b created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1493 +0x35f �[1mSTEP�[0m: Deleting pvc Jan 28 17:48:19.396: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comn5lbk" Jan 28 17:48:19.614: INFO: Waiting up to 3m0s for PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 to get deleted Jan 28 17:48:19.830: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (216.070561ms) Jan 28 17:48:25.046: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (5.43215515s) Jan 28 17:48:30.263: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (10.649332311s) Jan 28 17:48:35.482: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (15.867567184s) Jan 28 17:48:40.699: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (21.085200899s) Jan 28 17:48:45.918: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (26.303922298s) Jan 28 17:48:51.137: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (31.523313711s) Jan 28 17:48:56.353: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (36.739105877s) Jan 28 17:49:01.572: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (41.958208405s) Jan 28 17:49:06.792: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (47.177707459s) Jan 28 17:49:12.011: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (52.397085869s) Jan 28 17:49:17.231: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (57.617097397s) Jan 28 17:49:22.451: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (1m2.837025461s) Jan 28 17:49:27.667: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (1m8.053220398s) Jan 28 17:49:32.887: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (1m13.273021932s) Jan 28 17:49:38.111: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (1m18.497403552s) Jan 28 17:49:43.328: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (1m23.713650831s) Jan 28 17:49:48.545: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (1m28.930935389s) Jan 28 17:49:53.762: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (1m34.147836062s) Jan 28 17:49:58.978: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (1m39.363934916s) Jan 28 17:50:04.194: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (1m44.580222976s) Jan 28 17:50:09.411: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (1m49.79727655s) Jan 28 17:50:14.628: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (1m55.01400608s) Jan 28 17:50:19.844: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (2m0.230115687s) Jan 28 17:50:25.062: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (2m5.448340383s) Jan 28 17:50:30.279: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (2m10.664631934s) Jan 28 17:50:35.497: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (2m15.88267722s) Jan 28 17:50:40.715: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (2m21.100747211s) Jan 28 17:50:45.932: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (2m26.31778958s) Jan 28 17:50:51.151: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (2m31.537076916s) Jan 28 17:50:56.369: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (2m36.755240175s) Jan 28 17:51:01.586: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (2m41.972236964s) Jan 28 17:51:06.803: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (2m47.189036431s) Jan 28 17:51:12.019: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (2m52.405034447s) Jan 28 17:51:17.235: INFO: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 found and phase=Bound (2m57.621426785s) �[1mSTEP�[0m: Deleting sc Jan 28 17:51:22.458: FAIL: while cleanup resource Unexpected error: <errors.aggregate | len:1, cap:1>: [ [ { msg: "persistent Volume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 not deleted by dynamic provisioner: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 still exists within 3m0s", err: { s: "PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 still exists within 3m0s", }, }, ], ] persistent Volume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 not deleted by dynamic provisioner: PersistentVolume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 still exists within 3m0s occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func2() test/e2e/storage/testsuites/fsgroupchangepolicy.go:134 +0x24c panic({0x6ccb940, 0xc004562d00}) /usr/local/go/src/runtime/panic.go:884 +0x212 k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.Fail.func1() test/e2e/framework/ginkgowrapper/wrapper.go:63 +0x7d panic({0x63535e0, 0x7a2fa70}) /usr/local/go/src/runtime/panic.go:884 +0x212 k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.Fail({0xc003dd6140, 0x13b}, {0xc0026c4d28?, 0x7170289?, 0xc0026c4d48?}) test/e2e/framework/ginkgowrapper/wrapper.go:67 +0x197 k8s.io/kubernetes/test/e2e/framework.Fail({0xc003dd6000, 0x126}, {0xc002f7d820?, 0xc003dd6000?, 0xc001a8e1a0?}) test/e2e/framework/log.go:63 +0x145 k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).match(0xc0026c4e90, {0x7a5a5b8, 0xa705d80}, 0x0, {0x0, 0x0, 0x0}) vendor/github.com/onsi/gomega/internal/assertion/assertion.go:79 +0x1bd k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.(*Assertion).NotTo(0xc0026c4e90, {0x7a5a5b8, 0xa705d80}, {0x0, 0x0, 0x0}) vendor/github.com/onsi/gomega/internal/assertion/assertion.go:48 +0x92 k8s.io/kubernetes/test/e2e/framework.ExpectNoErrorWithOffset(0x7aa8fd8?, {0x7a339c0?, 0xc001bce8d0?}, {0x0?, 0x2?, 0x0?}) test/e2e/framework/expect.go:46 +0x9d k8s.io/kubernetes/test/e2e/framework.ExpectNoError(...) test/e2e/framework/expect.go:40 k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc002cbfb00, 0xc0026c5060, 0x0, {0xc001274538, 0x4}, {0xc00127453c, 0x4}) test/e2e/storage/testsuites/fsgroupchangepolicy.go:264 +0x151 k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() test/e2e/storage/testsuites/fsgroupchangepolicy.go:253 +0x556 k8s.io/kubernetes/test/e2e.RunE2ETests(0x25761d7?) test/e2e/e2e.go:130 +0x6bb k8s.io/kubernetes/test/e2e.TestE2E(0x24e52d9?) test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc00075d040, 0x741f9a8) /usr/local/go/src/testing/testing.go:1446 +0x10b created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1493 +0x35f [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy test/e2e/framework/framework.go:188 �[1mSTEP�[0m: Collecting events from namespace "fsgroupchangepolicy-7281". �[1mSTEP�[0m: Found 31 events. Jan 28 17:51:22.677: INFO: At 2023-01-28 17:40:07 +0000 UTC - event for ebs.csi.aws.comn5lbk: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Jan 28 17:51:22.677: INFO: At 2023-01-28 17:40:07 +0000 UTC - event for ebs.csi.aws.comn5lbk: {ebs.csi.aws.com_i-0550ecdc8b2275df1_a0fede8b-d858-4e18-b7bd-bd7186bb156f } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-7281/ebs.csi.aws.comn5lbk" Jan 28 17:51:22.677: INFO: At 2023-01-28 17:40:07 +0000 UTC - event for ebs.csi.aws.comn5lbk: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Jan 28 17:51:22.677: INFO: At 2023-01-28 17:40:11 +0000 UTC - event for ebs.csi.aws.comn5lbk: {ebs.csi.aws.com_i-0550ecdc8b2275df1_a0fede8b-d858-4e18-b7bd-bd7186bb156f } ProvisioningSucceeded: Successfully provisioned volume pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25 Jan 28 17:51:22.677: INFO: At 2023-01-28 17:40:11 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-7281/pod-4cc498ec-12fb-4bcf-a472-144d26af39ea to i-06d4799a5db582858 Jan 28 17:51:22.677: INFO: At 2023-01-28 17:40:14 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-f8ca6714-791e-41d4-93c8-2cf3f5ef6c25" Jan 28 17:51:22.677: INFO: At 2023-01-28 17:40:17 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0553798a29fb6893e4a7abd90c1751bac72d82e31a4a2e90457afb920d1fc26d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:40:30 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "79d6f627319396c34df4abe4db64ca5aef2a24f8eb2615b55500509bc37911a2": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:40:43 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c53ef978d5a4ef9da4e19d9a4d31cd0aad60ff38eb0dc030b9b356b21a587035": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:40:56 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "00d169e2c002dd7991b4951119f3c120f6c28ae4c07f9a1476b9f446615e1cb4": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:41:10 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4a9ca9b99fcceeebbf18b18d277a65b38cb5f05147613cc1f0b1e9bacbf00192": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:41:26 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e034f336870857a9740745378117c3fbc4e7ea9a0d9a7f6d7554a10229826be9": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:41:37 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fdf6833d9e42e1e72baf7ad45de870c5468f12e557244a522363f23c08bfc379": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:41:48 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f17abdf8cca2dc0c46faf71f3d15b3da58fe6952e1879a3f6c5cd2545043ae9b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:42:03 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5ab2a2dcc3711fafc744072150da180822a45cee8f51460f18ce342971c3da35": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:42:14 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "817656032a307dec70af089345e3635d7e3f4570bceba5f5181f5b9e495675cf": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:43:06 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {kubelet i-06d4799a5db582858} Created: Created container write-pod Jan 28 17:51:22.677: INFO: At 2023-01-28 17:43:06 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {kubelet i-06d4799a5db582858} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" already present on machine Jan 28 17:51:22.677: INFO: At 2023-01-28 17:43:06 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {kubelet i-06d4799a5db582858} Started: Started container write-pod Jan 28 17:51:22.677: INFO: At 2023-01-28 17:43:15 +0000 UTC - event for pod-4cc498ec-12fb-4bcf-a472-144d26af39ea: {kubelet i-06d4799a5db582858} Killing: Stopping container write-pod Jan 28 17:51:22.677: INFO: At 2023-01-28 17:43:18 +0000 UTC - event for pod-270e9e95-df4d-4169-9c6c-9329994f97a6: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-7281/pod-270e9e95-df4d-4169-9c6c-9329994f97a6 to i-06d4799a5db582858 Jan 28 17:51:22.677: INFO: At 2023-01-28 17:43:20 +0000 UTC - event for pod-270e9e95-df4d-4169-9c6c-9329994f97a6: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d64d167c82f255e33c98be45b43bdc093958d18391a6f403e476251d7632bdcd": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:43:35 +0000 UTC - event for pod-270e9e95-df4d-4169-9c6c-9329994f97a6: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d4c1547267b703f2a0f85c65ffeb383fa3e4b9456e92ddfeeeff9225f225c518": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:43:48 +0000 UTC - event for pod-270e9e95-df4d-4169-9c6c-9329994f97a6: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a80de9b3fcc46f592aa069020220113d02afc6b4006d4b03f7c7ed3d52620417": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:44:03 +0000 UTC - event for pod-270e9e95-df4d-4169-9c6c-9329994f97a6: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2c4e1847a8e638a005431312dbb6dc6439505034271c28ab2aafaca30dd652cd": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:44:14 +0000 UTC - event for pod-270e9e95-df4d-4169-9c6c-9329994f97a6: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2b5c3ceb5565f2fe6d938fb70fae0a5064b28b514f393f7d292d56527ccedc29": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:44:29 +0000 UTC - event for pod-270e9e95-df4d-4169-9c6c-9329994f97a6: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "33b00c737248dea2cb7f540c86e70f0d427f0fddc5f8ccc99e0ca34f603fd05e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:44:41 +0000 UTC - event for pod-270e9e95-df4d-4169-9c6c-9329994f97a6: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "500c761089b3db2c7c70729bd73e9ed9703f37d719b44fd58992050a4d042ac1": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:44:54 +0000 UTC - event for pod-270e9e95-df4d-4169-9c6c-9329994f97a6: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "800faf31a819ee0f0e0c4388f42543bce72235dbe82ab070f23764340b592e62": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:45:06 +0000 UTC - event for pod-270e9e95-df4d-4169-9c6c-9329994f97a6: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0ba22e0e35845525d0c6ea8701f2e4a835003eeae6b3d3e4deb47c84d964af5b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.677: INFO: At 2023-01-28 17:45:19 +0000 UTC - event for pod-270e9e95-df4d-4169-9c6c-9329994f97a6: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7e674f57b9962f40c4c5fc48c17155f2460876d85dbe3d86072e7a3f186dba72": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:22.893: INFO: POD NODE PHASE GRACE CONDITIONS Jan 28 17:51:22.893: INFO: pod-270e9e95-df4d-4169-9c6c-9329994f97a6 i-06d4799a5db582858 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:43:18 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:43:18 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:43:18 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:43:18 +0000 UTC }] Jan 28 17:51:22.893: INFO: Jan 28 17:51:23.111: INFO: Unable to fetch fsgroupchangepolicy-7281/pod-270e9e95-df4d-4169-9c6c-9329994f97a6/write-pod logs: the server rejected our request for an unknown reason (get pods pod-270e9e95-df4d-4169-9c6c-9329994f97a6) Jan 28 17:51:23.329: INFO: Logging node info for node i-03342b7c6ebddc1c6 Jan 28 17:51:23.544: INFO: Node Info: &Node{ObjectMeta:{i-03342b7c6ebddc1c6 2b79525a-8646-4141-9334-8d4a7a592265 36017 0 2023-01-28 17:22:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03342b7c6ebddc1c6 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-03342b7c6ebddc1c6 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.39.74 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03342b7c6ebddc1c6"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {e2e.test Update v1 2023-01-28 17:39:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:47:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-03342b7c6ebddc1c6,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.74,},NodeAddress{Type:ExternalIP,Address:52.221.218.125,},NodeAddress{Type:InternalDNS,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-218-125.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bb29c4b9b2fc4087668bfcd056585,SystemUUID:ec2bb29c-4b9b-2fc4-0876-68bfcd056585,BootID:042ee65a-a182-44df-9f74-80b116af3725,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:51:23.545: INFO: Logging kubelet events for node i-03342b7c6ebddc1c6 Jan 28 17:51:23.763: INFO: Logging pods the kubelet thinks is on node i-03342b7c6ebddc1c6 Jan 28 17:51:23.991: INFO: e2e-6bwsq-2vnvv started at 2023-01-28 17:45:45 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container c ready: false, restart count 0 Jan 28 17:51:23.992: INFO: ebs-csi-node-xxvv5 started at 2023-01-28 17:22:12 +0000 UTC (0+3 container statuses recorded) Jan 28 17:51:23.992: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:51:23.992: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:51:23.992: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:51:23.992: INFO: affinity-nodeport-2vxfn started at 2023-01-28 17:47:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container affinity-nodeport ready: false, restart count 0 Jan 28 17:51:23.992: INFO: echo-sourceip started at 2023-01-28 17:48:29 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container agnhost-container ready: false, restart count 0 Jan 28 17:51:23.992: INFO: pod-projected-configmaps-232ce7a1-e69f-4c43-88ec-9422d5087ff4 started at 2023-01-28 17:47:29 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container agnhost-container ready: false, restart count 0 Jan 28 17:51:23.992: INFO: httpd started at 2023-01-28 17:47:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container httpd ready: false, restart count 0 Jan 28 17:51:23.992: INFO: cilium-wlwtf started at 2023-01-28 17:22:12 +0000 UTC (1+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:51:23.992: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:51:23.992: INFO: slave started at 2023-01-28 17:46:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container cntr ready: false, restart count 0 Jan 28 17:51:23.992: INFO: netserver-0 started at 2023-01-28 17:50:55 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container webserver ready: false, restart count 0 Jan 28 17:51:23.992: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mkb65 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:51:23.992: INFO: pod-0 started at 2023-01-28 17:47:09 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container donothing ready: false, restart count 0 Jan 28 17:51:23.992: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-ch9br started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:51:23.992: INFO: master started at 2023-01-28 17:44:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container cntr ready: true, restart count 0 Jan 28 17:51:23.992: INFO: test-rollover-controller-nvj8x started at 2023-01-28 17:48:00 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container httpd ready: true, restart count 0 Jan 28 17:51:23.992: INFO: test-rollover-deployment-779c67f4f8-dgqvh started at 2023-01-28 17:50:55 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container agnhost ready: false, restart count 0 Jan 28 17:51:23.992: INFO: csi-hostpathplugin-0 started at 2023-01-28 17:50:24 +0000 UTC (0+7 container statuses recorded) Jan 28 17:51:23.992: INFO: Container csi-attacher ready: false, restart count 0 Jan 28 17:51:23.992: INFO: Container csi-provisioner ready: false, restart count 0 Jan 28 17:51:23.992: INFO: Container csi-resizer ready: false, restart count 0 Jan 28 17:51:23.992: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 28 17:51:23.992: INFO: Container hostpath ready: false, restart count 0 Jan 28 17:51:23.992: INFO: Container liveness-probe ready: false, restart count 0 Jan 28 17:51:23.992: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 28 17:51:23.992: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mnncj started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:51:23.992: INFO: terminate-cmd-rpn22cf6bad-1ba4-40d5-a5a0-310f21192352 started at 2023-01-28 17:50:09 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container terminate-cmd-rpn ready: false, restart count 0 Jan 28 17:51:23.992: INFO: test-rs-fl9dp started at 2023-01-28 17:50:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container httpd ready: false, restart count 0 Jan 28 17:51:23.992: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-57xvw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:23.992: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:51:24.724: INFO: Latency metrics for node i-03342b7c6ebddc1c6 Jan 28 17:51:24.724: INFO: Logging node info for node i-0550ecdc8b2275df1 Jan 28 17:51:24.940: INFO: Node Info: &Node{ObjectMeta:{i-0550ecdc8b2275df1 e2f1fe4c-911f-4da0-9481-51f08c3e654e 37029 0 2023-01-28 17:20:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0550ecdc8b2275df1 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.60.182 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0550ecdc8b2275df1"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:20:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-01-28 17:20:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-01-28 17:21:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-28 17:22:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0550ecdc8b2275df1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:21:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.182,},NodeAddress{Type:ExternalIP,Address:52.221.229.242,},NodeAddress{Type:InternalDNS,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-229-242.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec22a890f5274dd8ae5c60b32c44aae9,SystemUUID:ec22a890-f527-4dd8-ae5c-60b32c44aae9,BootID:83ea7fa2-7c97-4062-a52f-a856b154a02c,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:231502799,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64@sha256:6db2761074bae8385aa601d4b45ab065893cfdc258d8b2593fc56698c44dbee5 k8s.gcr.io/kube-apiserver-amd64:v1.24.10 registry.k8s.io/kube-apiserver-amd64:v1.24.10],SizeBytes:131733971,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64@sha256:8867df47c332bae8dfd061b4d4230cd0febcc5a8a427ecb2e8f70b33ffc6d732 k8s.gcr.io/kube-controller-manager-amd64:v1.24.10 registry.k8s.io/kube-controller-manager-amd64:v1.24.10],SizeBytes:121342265,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64@sha256:a589f10748fa227ea6ebf0730126f3f192cbdbb2c7f96a9d4685cabb86eb1cbc k8s.gcr.io/kube-scheduler-amd64:v1.24.10 registry.k8s.io/kube-scheduler-amd64:v1.24.10],SizeBytes:52751160,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.2],SizeBytes:43830714,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.2],SizeBytes:42821730,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:26802430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:245e81c2ed53b5508a6f687f3ab0c724b9ab31e6a5fa5e6d2f814981ab6ce55f registry.k8s.io/provider-aws/cloud-controller-manager:v1.24.3],SizeBytes:18282737,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.2],SizeBytes:4965809,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:51:24.940: INFO: Logging kubelet events for node i-0550ecdc8b2275df1 Jan 28 17:51:25.158: INFO: Logging pods the kubelet thinks is on node i-0550ecdc8b2275df1 Jan 28 17:51:25.382: INFO: kube-apiserver-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+2 container statuses recorded) Jan 28 17:51:25.382: INFO: Container healthcheck ready: true, restart count 0 Jan 28 17:51:25.382: INFO: Container kube-apiserver ready: true, restart count 0 Jan 28 17:51:25.382: INFO: kube-controller-manager-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:25.382: INFO: Container kube-controller-manager ready: true, restart count 3 Jan 28 17:51:25.382: INFO: kops-controller-gll72 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:25.382: INFO: Container kops-controller ready: true, restart count 0 Jan 28 17:51:25.382: INFO: cilium-4trt7 started at 2023-01-28 17:21:03 +0000 UTC (1+1 container statuses recorded) Jan 28 17:51:25.382: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:51:25.382: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:51:25.382: INFO: aws-cloud-controller-manager-w6p6f started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:25.382: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Jan 28 17:51:25.382: INFO: cilium-operator-8d8478c74-2vsdk started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:25.382: INFO: Container cilium-operator ready: true, restart count 0 Jan 28 17:51:25.382: INFO: etcd-manager-events-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:25.382: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:51:25.382: INFO: etcd-manager-main-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:25.382: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:51:25.382: INFO: ebs-csi-controller-779575c67d-kx2cv started at 2023-01-28 17:21:03 +0000 UTC (0+5 container statuses recorded) Jan 28 17:51:25.382: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:51:25.382: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:51:25.382: INFO: Container csi-resizer ready: true, restart count 0 Jan 28 17:51:25.382: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:51:25.382: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:51:25.382: INFO: dns-controller-7d4957564c-nd8s8 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:25.382: INFO: Container dns-controller ready: true, restart count 0 Jan 28 17:51:25.382: INFO: kube-scheduler-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:25.382: INFO: Container kube-scheduler ready: true, restart count 0 Jan 28 17:51:25.382: INFO: ebs-csi-node-stxgw started at 2023-01-28 17:21:03 +0000 UTC (0+3 container statuses recorded) Jan 28 17:51:25.382: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:51:25.382: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:51:25.382: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:51:26.081: INFO: Latency metrics for node i-0550ecdc8b2275df1 Jan 28 17:51:26.082: INFO: Logging node info for node i-06d4799a5db582858 Jan 28 17:51:26.297: INFO: Node Info: &Node{ObjectMeta:{i-06d4799a5db582858 aacf1bf6-d2bd-4f61-aad7-83531f5d9ba1 36980 0 2023-01-28 17:22:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-06d4799a5db582858 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-06d4799a5db582858 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.139 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06d4799a5db582858"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {e2e.test Update v1 2023-01-28 17:39:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:40:14 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 17:40:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-06d4799a5db582858,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.139,},NodeAddress{Type:ExternalIP,Address:18.143.137.61,},NodeAddress{Type:InternalDNS,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-143-137-61.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6f00e5a83cc422a6226665122e9d,SystemUUID:ec2e6f00-e5a8-3cc4-22a6-226665122e9d,BootID:3bfeb6d1-b807-4cfa-a0a7-954aef9e3823,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:49642095,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0870183487a8091e2 kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0870183487a8091e2,DevicePath:,},},Config:nil,},} Jan 28 17:51:26.298: INFO: Logging kubelet events for node i-06d4799a5db582858 Jan 28 17:51:26.519: INFO: Logging pods the kubelet thinks is on node i-06d4799a5db582858 Jan 28 17:51:26.750: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-r9lbh started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:51:26.750: INFO: e2e-6bwsq-8fx8x started at 2023-01-28 17:45:45 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container c ready: false, restart count 0 Jan 28 17:51:26.750: INFO: hostexec-i-06d4799a5db582858-4b8ml started at 2023-01-28 17:47:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:51:26.750: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-t9lsx started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:51:26.750: INFO: pod-subpath-test-inlinevolume-r6x2 started at 2023-01-28 17:50:10 +0000 UTC (0+2 container statuses recorded) Jan 28 17:51:26.750: INFO: Container test-container-subpath-inlinevolume-r6x2 ready: false, restart count 0 Jan 28 17:51:26.750: INFO: Container test-container-volume-inlinevolume-r6x2 ready: false, restart count 0 Jan 28 17:51:26.750: INFO: pod-12357688-a276-4eb3-96ce-6a76a486ef13 started at 2023-01-28 17:47:18 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:51:26.750: INFO: e2e-6bwsq-qxq2h started at 2023-01-28 17:47:56 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container c ready: false, restart count 0 Jan 28 17:51:26.750: INFO: security-context-03128938-2d13-461d-9fc8-c34cea426a4f started at 2023-01-28 17:51:05 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container test-container ready: false, restart count 0 Jan 28 17:51:26.750: INFO: cilium-8cb5h started at 2023-01-28 17:22:14 +0000 UTC (1+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:51:26.750: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:51:26.750: INFO: affinity-nodeport-9d4b5 started at 2023-01-28 17:47:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container affinity-nodeport ready: true, restart count 0 Jan 28 17:51:26.750: INFO: local-injector started at 2023-01-28 17:47:20 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container local-injector ready: false, restart count 0 Jan 28 17:51:26.750: INFO: e2e-6bwsq-7dz2j started at 2023-01-28 17:46:22 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container c ready: false, restart count 0 Jan 28 17:51:26.750: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-lw565 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:51:26.750: INFO: ephemeral-containers-target-pod started at 2023-01-28 17:47:14 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container test-container-1 ready: false, restart count 0 Jan 28 17:51:26.750: INFO: ebs-csi-node-ttfmk started at 2023-01-28 17:22:14 +0000 UTC (0+3 container statuses recorded) Jan 28 17:51:26.750: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:51:26.750: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:51:26.750: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:51:26.750: INFO: pod-270e9e95-df4d-4169-9c6c-9329994f97a6 started at 2023-01-28 17:43:18 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:51:26.750: INFO: test-rs-2cpll started at 2023-01-28 17:45:37 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container httpd ready: true, restart count 0 Jan 28 17:51:26.750: INFO: netserver-1 started at 2023-01-28 17:50:55 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container webserver ready: false, restart count 0 Jan 28 17:51:26.750: INFO: pod-1 started at 2023-01-28 17:47:09 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container donothing ready: false, restart count 0 Jan 28 17:51:26.750: INFO: termination-message-containera2a63b81-5c48-4c6b-963e-ce633502def0 started at 2023-01-28 17:51:19 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:26.750: INFO: Container termination-message-container ready: false, restart count 0 Jan 28 17:51:27.618: INFO: Latency metrics for node i-06d4799a5db582858 Jan 28 17:51:27.618: INFO: Logging node info for node i-0d3b431203a4a372c Jan 28 17:51:27.834: INFO: Node Info: &Node{ObjectMeta:{i-0d3b431203a4a372c b69212ba-b617-41e9-91e3-ee5929b727b2 38143 0 2023-01-28 17:22:12 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a io.kubernetes.storage.mock/node:some-mock-node kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d3b431203a4a372c kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0d3b431203a4a372c topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.33.52 csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-151":"i-0d3b431203a4a372c","ebs.csi.aws.com":"i-0d3b431203a4a372c"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {e2e.test Update v1 2023-01-28 17:39:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:45:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0d3b431203a4a372c,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:28 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:28 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:28 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:50:28 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.52,},NodeAddress{Type:ExternalIP,Address:13.229.100.227,},NodeAddress{Type:InternalDNS,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-229-100-227.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2fe0eb6b322558c1427d0b9ac3c84d,SystemUUID:ec2fe0eb-6b32-2558-c142-7d0b9ac3c84d,BootID:47c658c8-8593-40bf-a02e-622027452e1a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:51:27.835: INFO: Logging kubelet events for node i-0d3b431203a4a372c Jan 28 17:51:28.053: INFO: Logging pods the kubelet thinks is on node i-0d3b431203a4a372c Jan 28 17:51:28.276: INFO: cilium-cs2hk started at 2023-01-28 17:22:13 +0000 UTC (1+1 container statuses recorded) Jan 28 17:51:28.278: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:51:28.278: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:51:28.278: INFO: netserver-2 started at 2023-01-28 17:50:55 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:28.278: INFO: Container webserver ready: false, restart count 0 Jan 28 17:51:28.278: INFO: csi-mockplugin-0 started at 2023-01-28 17:47:23 +0000 UTC (0+3 container statuses recorded) Jan 28 17:51:28.278: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:51:28.278: INFO: Container driver-registrar ready: true, restart count 0 Jan 28 17:51:28.278: INFO: Container mock ready: true, restart count 0 Jan 28 17:51:28.278: INFO: pod-with-poststart-exec-hook started at 2023-01-28 17:51:26 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:28.278: INFO: Container pod-with-poststart-exec-hook ready: false, restart count 0 Jan 28 17:51:28.278: INFO: pvc-volume-tester-9brq4 started at 2023-01-28 17:51:14 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:28.278: INFO: Container volume-tester ready: false, restart count 0 Jan 28 17:51:28.278: INFO: csi-mockplugin-resizer-0 started at 2023-01-28 17:47:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:28.278: INFO: Container csi-resizer ready: false, restart count 0 Jan 28 17:51:28.278: INFO: test-rs-sr5rf started at 2023-01-28 17:50:24 +0000 UTC (0+2 container statuses recorded) Jan 28 17:51:28.278: INFO: Container httpd ready: false, restart count 0 Jan 28 17:51:28.278: INFO: Container test-rs ready: false, restart count 0 Jan 28 17:51:28.278: INFO: ebs-csi-node-j2pwn started at 2023-01-28 17:22:13 +0000 UTC (0+3 container statuses recorded) Jan 28 17:51:28.278: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:51:28.278: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:51:28.278: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:51:28.278: INFO: coredns-6dd78b7459-t8lf4 started at 2023-01-28 17:22:48 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:28.278: INFO: Container coredns ready: true, restart count 0 Jan 28 17:51:28.278: INFO: hostexec-i-0d3b431203a4a372c-5n26x started at 2023-01-28 17:50:27 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:28.278: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:51:28.278: INFO: pod-2934e8e9-9ba5-4f70-a4c1-3be753776a5f started at 2023-01-28 17:50:33 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:28.278: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:51:29.175: INFO: Latency metrics for node i-0d3b431203a4a372c Jan 28 17:51:29.175: INFO: Logging node info for node i-0e10813507581b097 Jan 28 17:51:29.391: INFO: Node Info: &Node{ObjectMeta:{i-0e10813507581b097 8af9c758-a6f6-4a0d-b634-4d99ed71b929 37803 0 2023-01-28 17:22:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e10813507581b097 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0e10813507581b097 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.58 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e10813507581b097"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {e2e.test Update v1 2023-01-28 17:39:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:40:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0e10813507581b097,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:34 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:34 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:34 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:50:34 +0000 UTC,LastTransitionTime:2023-01-28 17:22:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.58,},NodeAddress{Type:ExternalIP,Address:54.254.187.204,},NodeAddress{Type:InternalDNS,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-187-204.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2922b3ab12fd264b764b33da3e62c2,SystemUUID:ec2922b3-ab12-fd26-4b76-4b33da3e62c2,BootID:07cb73db-252f-45aa-8a98-af3a42c0313b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:51:29.392: INFO: Logging kubelet events for node i-0e10813507581b097 Jan 28 17:51:29.609: INFO: Logging pods the kubelet thinks is on node i-0e10813507581b097 Jan 28 17:51:29.831: INFO: boom-server started at 2023-01-28 17:48:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:29.831: INFO: Container boom-server ready: false, restart count 0 Jan 28 17:51:29.831: INFO: pause-pod-596cfff4b4-b25wr started at <nil> (0+0 container statuses recorded) Jan 28 17:51:29.831: INFO: ebs-csi-node-k6rdn started at 2023-01-28 17:22:08 +0000 UTC (0+3 container statuses recorded) Jan 28 17:51:29.831: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:51:29.831: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:51:29.831: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:51:29.831: INFO: affinity-nodeport-2mdr9 started at 2023-01-28 17:47:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:29.831: INFO: Container affinity-nodeport ready: false, restart count 0 Jan 28 17:51:29.831: INFO: netserver-3 started at 2023-01-28 17:50:55 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:29.831: INFO: Container webserver ready: false, restart count 0 Jan 28 17:51:29.831: INFO: pod-sharedvolume-6436a328-e6e9-4d09-9aaa-d90e803e79b6 started at <nil> (0+0 container statuses recorded) Jan 28 17:51:29.831: INFO: coredns-6dd78b7459-qk7rr started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:29.831: INFO: Container coredns ready: true, restart count 0 Jan 28 17:51:29.831: INFO: coredns-autoscaler-5f76bf6c7b-nnmkq started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:29.831: INFO: Container autoscaler ready: true, restart count 0 Jan 28 17:51:29.831: INFO: pod-handle-http-request started at 2023-01-28 17:50:51 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:29.831: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:51:29.831: INFO: cilium-mbfkc started at 2023-01-28 17:22:08 +0000 UTC (1+1 container statuses recorded) Jan 28 17:51:29.831: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:51:29.831: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:51:30.595: INFO: Latency metrics for node i-0e10813507581b097 Jan 28 17:51:30.595: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "fsgroupchangepolicy-7281" for this suite.
Find pod-270e9e95-df4d-4169-9c6c-9329994f97a6 mentions in log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-cli\]\sKubectl\sclient\sSimple\spod\sshould\sreturn\scommand\sexit\scodes\srunning\sa\ssuccessful\scommand$'
test/e2e/kubectl/kubectl.go:382 Jan 28 17:44:56.316: Expected <bool>: false to equal <bool>: true test/e2e/kubectl/kubectl.go:386from junit_16.xml
[BeforeEach] [sig-cli] Kubectl client test/e2e/framework/framework.go:187 �[1mSTEP�[0m: Creating a kubernetes client Jan 28 17:39:51.297: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename kubectl �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-cli] Kubectl client test/e2e/kubectl/kubectl.go:245 [BeforeEach] Simple pod test/e2e/kubectl/kubectl.go:382 �[1mSTEP�[0m: creating the pod from Jan 28 17:39:52.801: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-1169 create -f -' Jan 28 17:39:55.019: INFO: stderr: "" Jan 28 17:39:55.019: INFO: stdout: "pod/httpd created\n" Jan 28 17:39:55.019: INFO: Waiting up to 5m0s for 1 pods to be running and ready: [httpd] Jan 28 17:39:55.019: INFO: Waiting up to 5m0s for pod "httpd" in namespace "kubectl-1169" to be "running and ready" Jan 28 17:39:55.234: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 215.119376ms Jan 28 17:39:57.451: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2.431757603s Jan 28 17:39:59.667: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4.647588112s Jan 28 17:40:01.883: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 6.864176227s Jan 28 17:40:04.099: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 9.079491037s Jan 28 17:40:06.314: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 11.295142541s Jan 28 17:40:08.530: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 13.510857357s Jan 28 17:40:10.746: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 15.726381092s Jan 28 17:40:12.960: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 17.940699904s Jan 28 17:40:15.176: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 20.156795541s Jan 28 17:40:17.390: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 22.371295956s Jan 28 17:40:19.606: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 24.58702136s Jan 28 17:40:21.822: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 26.802766315s Jan 28 17:40:24.037: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 29.017909194s Jan 28 17:40:26.253: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 31.233742108s Jan 28 17:40:28.469: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 33.44960836s Jan 28 17:40:30.685: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 35.66551401s Jan 28 17:40:32.900: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 37.880940525s Jan 28 17:40:35.115: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 40.095471858s Jan 28 17:40:37.330: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 42.310863961s Jan 28 17:40:39.545: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 44.525545192s Jan 28 17:40:41.759: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 46.739974364s Jan 28 17:40:43.974: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 48.954589065s Jan 28 17:40:46.188: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 51.168870603s Jan 28 17:40:48.403: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 53.383857754s Jan 28 17:40:50.620: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 55.600938194s Jan 28 17:40:52.835: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 57.816075669s Jan 28 17:40:55.050: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.030948705s Jan 28 17:40:57.266: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.246690633s Jan 28 17:40:59.482: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.462429425s Jan 28 17:41:01.697: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.677393328s Jan 28 17:41:03.912: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.892406058s Jan 28 17:41:06.126: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m11.107213565s Jan 28 17:41:08.342: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m13.322774949s Jan 28 17:41:10.558: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m15.538484819s Jan 28 17:41:12.773: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m17.753916508s Jan 28 17:41:14.989: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m19.969789951s Jan 28 17:41:17.204: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.184783394s Jan 28 17:41:19.419: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.399928978s Jan 28 17:41:21.635: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.615411033s Jan 28 17:41:23.850: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.831142165s Jan 28 17:41:26.066: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m31.046427597s Jan 28 17:41:28.281: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m33.261394574s Jan 28 17:41:30.497: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m35.47762395s Jan 28 17:41:32.712: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m37.692945611s Jan 28 17:41:34.929: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m39.909859745s Jan 28 17:41:37.145: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m42.125870593s Jan 28 17:41:39.360: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.341261201s Jan 28 17:41:41.576: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.556868591s Jan 28 17:41:43.791: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.772195656s Jan 28 17:41:46.007: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m50.988176774s Jan 28 17:41:48.222: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m53.203319435s Jan 28 17:41:50.437: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m55.417654119s Jan 28 17:41:52.653: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m57.633792234s Jan 28 17:41:54.868: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 1m59.848638418s Jan 28 17:41:57.085: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m2.065435551s Jan 28 17:41:59.300: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m4.280529006s Jan 28 17:42:01.516: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.496850713s Jan 28 17:42:03.731: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.712163667s Jan 28 17:42:05.947: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.927738128s Jan 28 17:42:08.162: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m13.142804629s Jan 28 17:42:10.376: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m15.357320943s Jan 28 17:42:12.591: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m17.572014689s Jan 28 17:42:14.806: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m19.786671451s Jan 28 17:42:17.023: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.004129182s Jan 28 17:42:19.238: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.218982641s Jan 28 17:42:21.454: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.434878195s Jan 28 17:42:23.669: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.649724506s Jan 28 17:42:25.883: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.864298577s Jan 28 17:42:28.099: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m33.079983842s Jan 28 17:42:30.313: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m35.294206502s Jan 28 17:42:32.529: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m37.509930425s Jan 28 17:42:34.745: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m39.726002629s Jan 28 17:42:36.961: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m41.942101752s Jan 28 17:42:39.177: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.157384064s Jan 28 17:42:41.392: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.372826643s Jan 28 17:42:43.607: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.587541334s Jan 28 17:42:45.822: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.803029578s Jan 28 17:42:48.038: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m53.018927457s Jan 28 17:42:50.254: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m55.234433159s Jan 28 17:42:52.468: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m57.449213676s Jan 28 17:42:54.684: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 2m59.665358252s Jan 28 17:42:56.900: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m1.880875573s Jan 28 17:42:59.116: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.096458178s Jan 28 17:43:01.331: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m6.311491162s Jan 28 17:43:03.545: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m8.52585659s Jan 28 17:43:05.760: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.740795717s Jan 28 17:43:07.975: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.955986919s Jan 28 17:43:10.190: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m15.170656574s Jan 28 17:43:12.405: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m17.385655663s Jan 28 17:43:14.621: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m19.601787416s Jan 28 17:43:16.837: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m21.817476047s Jan 28 17:43:19.052: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.032578346s Jan 28 17:43:21.268: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.249116607s Jan 28 17:43:23.483: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.463932149s Jan 28 17:43:25.698: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.679363922s Jan 28 17:43:27.913: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.894223589s Jan 28 17:43:30.129: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m35.11012168s Jan 28 17:43:32.344: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m37.324727351s Jan 28 17:43:34.559: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m39.53992968s Jan 28 17:43:36.775: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m41.755747313s Jan 28 17:43:38.990: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m43.970419585s Jan 28 17:43:41.205: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m46.186242915s Jan 28 17:43:43.420: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m48.401166583s Jan 28 17:43:45.637: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m50.617459982s Jan 28 17:43:47.853: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m52.833435372s Jan 28 17:43:50.068: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m55.048426793s Jan 28 17:43:52.284: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m57.264494838s Jan 28 17:43:54.499: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 3m59.479403085s Jan 28 17:43:56.714: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m1.695130701s Jan 28 17:43:58.930: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m3.910567235s Jan 28 17:44:01.144: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.125324942s Jan 28 17:44:03.362: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.342640297s Jan 28 17:44:05.576: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.557219084s Jan 28 17:44:07.792: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.773223264s Jan 28 17:44:10.008: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.988955355s Jan 28 17:44:12.223: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m17.203861762s Jan 28 17:44:14.438: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m19.418780831s Jan 28 17:44:16.653: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m21.634222922s Jan 28 17:44:18.868: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m23.84878782s Jan 28 17:44:21.085: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.065603481s Jan 28 17:44:23.300: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.280662885s Jan 28 17:44:25.515: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.495658073s Jan 28 17:44:27.730: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.711324186s Jan 28 17:44:29.946: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.926473369s Jan 28 17:44:32.161: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m37.142210926s Jan 28 17:44:34.377: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m39.357387254s Jan 28 17:44:36.592: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m41.572896489s Jan 28 17:44:38.807: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m43.787498333s Jan 28 17:44:41.022: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.002761657s Jan 28 17:44:43.237: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.217882718s Jan 28 17:44:45.453: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.433509261s Jan 28 17:44:47.667: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.64829409s Jan 28 17:44:49.882: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.862733281s Jan 28 17:44:52.098: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.078836466s Jan 28 17:44:54.314: INFO: Pod "httpd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.294708824s Jan 28 17:44:56.316: INFO: Pod httpd failed to be running and ready. Jan 28 17:44:56.316: INFO: Wanted all 1 pods to be running and ready. Result: false. Pods: [httpd] Jan 28 17:44:56.316: FAIL: Expected <bool>: false to equal <bool>: true Full Stack Trace k8s.io/kubernetes/test/e2e/kubectl.glob..func1.8.1() test/e2e/kubectl/kubectl.go:386 +0x245 k8s.io/kubernetes/test/e2e.RunE2ETests(0x25761d7?) test/e2e/e2e.go:130 +0x6bb k8s.io/kubernetes/test/e2e.TestE2E(0x24e52d9?) test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc00080bba0, 0x741f9a8) /usr/local/go/src/testing/testing.go:1446 +0x10b created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1493 +0x35f [AfterEach] Simple pod test/e2e/kubectl/kubectl.go:388 �[1mSTEP�[0m: using delete to clean up resources Jan 28 17:44:56.317: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-1169 delete --grace-period=0 --force -f -' Jan 28 17:44:57.285: INFO: stderr: "warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.\n" Jan 28 17:44:57.285: INFO: stdout: "pod \"httpd\" force deleted\n" Jan 28 17:44:57.285: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-1169 get rc,svc -l name=httpd --no-headers' Jan 28 17:44:58.242: INFO: stderr: "No resources found in kubectl-1169 namespace.\n" Jan 28 17:44:58.242: INFO: stdout: "" Jan 28 17:44:58.242: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-1169 get pods -l name=httpd -o go-template={{ range .items }}{{ if not .metadata.deletionTimestamp }}{{ .metadata.name }}{{ "\n" }}{{ end }}{{ end }}' Jan 28 17:44:58.975: INFO: stderr: "" Jan 28 17:44:58.975: INFO: stdout: "" [AfterEach] [sig-cli] Kubectl client test/e2e/framework/framework.go:188 �[1mSTEP�[0m: Collecting events from namespace "kubectl-1169". �[1mSTEP�[0m: Found 11 events. Jan 28 17:44:59.190: INFO: At 2023-01-28 17:39:54 +0000 UTC - event for httpd: {default-scheduler } Scheduled: Successfully assigned kubectl-1169/httpd to i-03342b7c6ebddc1c6 Jan 28 17:44:59.190: INFO: At 2023-01-28 17:39:55 +0000 UTC - event for httpd: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f1e8d7e051f62957809f048d6aa766ff13ead78189af47fc362741e6698925f5": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:59.190: INFO: At 2023-01-28 17:40:08 +0000 UTC - event for httpd: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ccab2301c886a9be1dd4afd297140ceb6e13ffff23aa3eaca720f3aa13307b50": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:59.190: INFO: At 2023-01-28 17:40:22 +0000 UTC - event for httpd: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ef05cffedfd04dc037cd5daf676a4408e78a7bfd96bbcf7f611687681923deaa": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:59.190: INFO: At 2023-01-28 17:40:35 +0000 UTC - event for httpd: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c7ca84981d5f71b8a789f413b2b889aaed6807bcc1dd0e6443428734b4828907": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:59.190: INFO: At 2023-01-28 17:40:46 +0000 UTC - event for httpd: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "53cc889e345c793d605300801919135911160ce52a1ccfd84b1e5bc784a96813": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:59.190: INFO: At 2023-01-28 17:40:57 +0000 UTC - event for httpd: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6112f14555722c4e438af5e7b2ec2bfd8bb6c270bdad68d3cdf9703d051d953a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:59.190: INFO: At 2023-01-28 17:41:08 +0000 UTC - event for httpd: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "84e149a7a4fd35eb685eb99dfce1394eb3575a02a4713ba2ea8cf67657483085": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:59.190: INFO: At 2023-01-28 17:41:20 +0000 UTC - event for httpd: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ded01a8f5ea678ddb214e6de0aa1dfa283aeb74a23c64e426440c4d4710d319c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:59.190: INFO: At 2023-01-28 17:41:32 +0000 UTC - event for httpd: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "43ec8e652da52bcae9724e364e0e3baf480062239c652867ee2b614b98e9f969": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:59.190: INFO: At 2023-01-28 17:41:46 +0000 UTC - event for httpd: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a2718068227fc10e1a9dba01cca3d67d9aee92dc1a961178cb2a1288afc43f0e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:59.404: INFO: POD NODE PHASE GRACE CONDITIONS Jan 28 17:44:59.404: INFO: Jan 28 17:44:59.621: INFO: Logging node info for node i-03342b7c6ebddc1c6 Jan 28 17:44:59.835: INFO: Node Info: &Node{ObjectMeta:{i-03342b7c6ebddc1c6 2b79525a-8646-4141-9334-8d4a7a592265 32812 0 2023-01-28 17:22:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03342b7c6ebddc1c6 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-03342b7c6ebddc1c6 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.39.74 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03342b7c6ebddc1c6"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:37:15 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 17:37:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {e2e.test Update v1 2023-01-28 17:39:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-03342b7c6ebddc1c6,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:13 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:13 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:13 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:40:13 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.74,},NodeAddress{Type:ExternalIP,Address:52.221.218.125,},NodeAddress{Type:InternalDNS,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-218-125.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bb29c4b9b2fc4087668bfcd056585,SystemUUID:ec2bb29c-4b9b-2fc4-0876-68bfcd056585,BootID:042ee65a-a182-44df-9f74-80b116af3725,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-02c58416f403003e0],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02c58416f403003e0,DevicePath:,},},Config:nil,},} Jan 28 17:44:59.836: INFO: Logging kubelet events for node i-03342b7c6ebddc1c6 Jan 28 17:45:00.052: INFO: Logging pods the kubelet thinks is on node i-03342b7c6ebddc1c6 Jan 28 17:45:00.280: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-57xvw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:00.280: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-kcssq started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:00.280: INFO: up-down-2-rw86m started at 2023-01-28 17:38:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Container up-down-2 ready: true, restart count 0 Jan 28 17:45:00.280: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-jqzfn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:00.280: INFO: pod-secrets-49e03bcd-e201-4e96-8caa-cef1cb071d5d started at 2023-01-28 17:40:18 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Container secret-volume-test ready: false, restart count 0 Jan 28 17:45:00.280: INFO: ebs-csi-node-xxvv5 started at 2023-01-28 17:22:12 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:00.280: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:00.280: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:00.280: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:45:00.280: INFO: up-down-3-rcvxq started at 2023-01-28 17:39:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Container up-down-3 ready: true, restart count 0 Jan 28 17:45:00.280: INFO: startup-f07e9592-0502-4a24-aff9-ef1ad4288606 started at 2023-01-28 17:39:59 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Container busybox ready: false, restart count 0 Jan 28 17:45:00.280: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6vfpn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:00.280: INFO: pod-7824b9b1-2ebe-4f4f-80f3-fe86d5163ca8 started at 2023-01-28 17:40:09 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Container test-container ready: false, restart count 0 Jan 28 17:45:00.280: INFO: cilium-wlwtf started at 2023-01-28 17:22:12 +0000 UTC (1+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:45:00.280: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:45:00.280: INFO: hostexec-i-03342b7c6ebddc1c6-s4cmv started at 2023-01-28 17:44:49 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:45:00.280: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mkb65 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:00.280: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-9jldw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:00.280: INFO: master started at 2023-01-28 17:44:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Container cntr ready: false, restart count 0 Jan 28 17:45:00.280: INFO: pod-d024b2fd-b2af-4707-9028-d6c200999384 started at 2023-01-28 17:44:57 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:45:00.280: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-fgg6g started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.280: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:00.281: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-ch9br started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.281: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:00.281: INFO: test-ss-1 started at 2023-01-28 17:40:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.281: INFO: Container webserver ready: false, restart count 0 Jan 28 17:45:00.281: INFO: pod-23984187-216b-4e72-b531-729449648f82 started at 2023-01-28 17:43:00 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.281: INFO: Container test-container ready: false, restart count 0 Jan 28 17:45:00.281: INFO: update-demo-nautilus-vl9xg started at 2023-01-28 17:44:51 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.281: INFO: Container update-demo ready: false, restart count 0 Jan 28 17:45:00.281: INFO: ss-1 started at 2023-01-28 17:39:40 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.281: INFO: Container webserver ready: false, restart count 0 Jan 28 17:45:00.281: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mnncj started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.281: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:00.281: INFO: adopt-release-56c4c started at 2023-01-28 17:42:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.281: INFO: Container c ready: false, restart count 0 Jan 28 17:45:00.281: INFO: rs-4ztpt started at 2023-01-28 17:40:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.281: INFO: Container donothing ready: true, restart count 0 Jan 28 17:45:00.281: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6b7pn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:00.281: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:01.200: INFO: Latency metrics for node i-03342b7c6ebddc1c6 Jan 28 17:45:01.200: INFO: Logging node info for node i-0550ecdc8b2275df1 Jan 28 17:45:01.415: INFO: Node Info: &Node{ObjectMeta:{i-0550ecdc8b2275df1 e2f1fe4c-911f-4da0-9481-51f08c3e654e 33948 0 2023-01-28 17:20:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0550ecdc8b2275df1 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.60.182 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0550ecdc8b2275df1"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:20:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-01-28 17:20:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-01-28 17:21:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-28 17:22:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0550ecdc8b2275df1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:21:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.182,},NodeAddress{Type:ExternalIP,Address:52.221.229.242,},NodeAddress{Type:InternalDNS,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-229-242.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec22a890f5274dd8ae5c60b32c44aae9,SystemUUID:ec22a890-f527-4dd8-ae5c-60b32c44aae9,BootID:83ea7fa2-7c97-4062-a52f-a856b154a02c,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:231502799,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64@sha256:6db2761074bae8385aa601d4b45ab065893cfdc258d8b2593fc56698c44dbee5 k8s.gcr.io/kube-apiserver-amd64:v1.24.10 registry.k8s.io/kube-apiserver-amd64:v1.24.10],SizeBytes:131733971,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64@sha256:8867df47c332bae8dfd061b4d4230cd0febcc5a8a427ecb2e8f70b33ffc6d732 k8s.gcr.io/kube-controller-manager-amd64:v1.24.10 registry.k8s.io/kube-controller-manager-amd64:v1.24.10],SizeBytes:121342265,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64@sha256:a589f10748fa227ea6ebf0730126f3f192cbdbb2c7f96a9d4685cabb86eb1cbc k8s.gcr.io/kube-scheduler-amd64:v1.24.10 registry.k8s.io/kube-scheduler-amd64:v1.24.10],SizeBytes:52751160,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.2],SizeBytes:43830714,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.2],SizeBytes:42821730,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:26802430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:245e81c2ed53b5508a6f687f3ab0c724b9ab31e6a5fa5e6d2f814981ab6ce55f registry.k8s.io/provider-aws/cloud-controller-manager:v1.24.3],SizeBytes:18282737,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.2],SizeBytes:4965809,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:45:01.415: INFO: Logging kubelet events for node i-0550ecdc8b2275df1 Jan 28 17:45:01.632: INFO: Logging pods the kubelet thinks is on node i-0550ecdc8b2275df1 Jan 28 17:45:01.855: INFO: kube-scheduler-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:01.855: INFO: Container kube-scheduler ready: true, restart count 0 Jan 28 17:45:01.855: INFO: ebs-csi-node-stxgw started at 2023-01-28 17:21:03 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:01.855: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:01.855: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:01.855: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:45:01.855: INFO: dns-controller-7d4957564c-nd8s8 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:01.855: INFO: Container dns-controller ready: true, restart count 0 Jan 28 17:45:01.855: INFO: kube-controller-manager-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:01.855: INFO: Container kube-controller-manager ready: true, restart count 3 Jan 28 17:45:01.855: INFO: kops-controller-gll72 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:01.855: INFO: Container kops-controller ready: true, restart count 0 Jan 28 17:45:01.855: INFO: cilium-4trt7 started at 2023-01-28 17:21:03 +0000 UTC (1+1 container statuses recorded) Jan 28 17:45:01.855: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:45:01.855: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:45:01.855: INFO: aws-cloud-controller-manager-w6p6f started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:01.855: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Jan 28 17:45:01.855: INFO: cilium-operator-8d8478c74-2vsdk started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:01.855: INFO: Container cilium-operator ready: true, restart count 0 Jan 28 17:45:01.855: INFO: etcd-manager-events-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:01.855: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:45:01.855: INFO: etcd-manager-main-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:01.855: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:45:01.855: INFO: kube-apiserver-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+2 container statuses recorded) Jan 28 17:45:01.855: INFO: Container healthcheck ready: true, restart count 0 Jan 28 17:45:01.855: INFO: Container kube-apiserver ready: true, restart count 0 Jan 28 17:45:01.855: INFO: ebs-csi-controller-779575c67d-kx2cv started at 2023-01-28 17:21:03 +0000 UTC (0+5 container statuses recorded) Jan 28 17:45:01.855: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:45:01.855: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:45:01.855: INFO: Container csi-resizer ready: true, restart count 0 Jan 28 17:45:01.855: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:01.855: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:02.562: INFO: Latency metrics for node i-0550ecdc8b2275df1 Jan 28 17:45:02.562: INFO: Logging node info for node i-06d4799a5db582858 Jan 28 17:45:02.777: INFO: Node Info: &Node{ObjectMeta:{i-06d4799a5db582858 aacf1bf6-d2bd-4f61-aad7-83531f5d9ba1 33004 0 2023-01-28 17:22:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-06d4799a5db582858 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-06d4799a5db582858 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.139 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06d4799a5db582858"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {e2e.test Update v1 2023-01-28 17:39:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:40:14 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 17:40:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-06d4799a5db582858,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.139,},NodeAddress{Type:ExternalIP,Address:18.143.137.61,},NodeAddress{Type:InternalDNS,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-143-137-61.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6f00e5a83cc422a6226665122e9d,SystemUUID:ec2e6f00-e5a8-3cc4-22a6-226665122e9d,BootID:3bfeb6d1-b807-4cfa-a0a7-954aef9e3823,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc kubernetes.io/csi/ebs.csi.aws.com^vol-0dff02acc75d59bc3],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0dff02acc75d59bc3,DevicePath:,},},Config:nil,},} Jan 28 17:45:02.778: INFO: Logging kubelet events for node i-06d4799a5db582858 Jan 28 17:45:02.995: INFO: Logging pods the kubelet thinks is on node i-06d4799a5db582858 Jan 28 17:45:03.221: INFO: pod-configmaps-7cd33bfb-d4f5-467e-946d-cdf07018d8cf started at 2023-01-28 17:43:01 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container env-test ready: false, restart count 0 Jan 28 17:45:03.222: INFO: httpd started at 2023-01-28 17:44:35 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container httpd ready: false, restart count 0 Jan 28 17:45:03.222: INFO: up-down-2-vg8d2 started at 2023-01-28 17:38:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container up-down-2 ready: true, restart count 0 Jan 28 17:45:03.222: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-dgjbr started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:03.222: INFO: cilium-8cb5h started at 2023-01-28 17:22:14 +0000 UTC (1+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:45:03.222: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:45:03.222: INFO: ss-0 started at 2023-01-28 17:40:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container webserver ready: false, restart count 0 Jan 28 17:45:03.222: INFO: update-demo-nautilus-zqp4x started at 2023-01-28 17:44:51 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container update-demo ready: false, restart count 0 Jan 28 17:45:03.222: INFO: csi-mockplugin-0 started at 2023-01-28 17:40:55 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:03.222: INFO: Container csi-provisioner ready: false, restart count 0 Jan 28 17:45:03.222: INFO: Container driver-registrar ready: false, restart count 0 Jan 28 17:45:03.222: INFO: Container mock ready: false, restart count 0 Jan 28 17:45:03.222: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-bdtqt started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:03.222: INFO: pod-server-1 started at 2023-01-28 17:40:22 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container agnhost-container ready: false, restart count 0 Jan 28 17:45:03.222: INFO: adopt-release-g5t8f started at 2023-01-28 17:42:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container c ready: false, restart count 0 Jan 28 17:45:03.222: INFO: ebs-csi-node-ttfmk started at 2023-01-28 17:22:14 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:03.222: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:03.222: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:03.222: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:45:03.222: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-lw565 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:03.222: INFO: busybox-702a2763-bdb2-44b8-ae2d-94e4cd0026af started at 2023-01-28 17:42:05 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container busybox ready: false, restart count 0 Jan 28 17:45:03.222: INFO: rs-tbjch started at 2023-01-28 17:40:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container donothing ready: false, restart count 0 Jan 28 17:45:03.222: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-jq2xw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:03.222: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-7dx5z started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:03.222: INFO: pod-270e9e95-df4d-4169-9c6c-9329994f97a6 started at 2023-01-28 17:43:18 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:45:03.222: INFO: up-down-3-r87lj started at 2023-01-28 17:39:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container up-down-3 ready: true, restart count 0 Jan 28 17:45:03.222: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-t9lsx started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:03.222: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-r9lbh started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:03.222: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-slm8r started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:03.222: INFO: emptydir-injector started at 2023-01-28 17:40:31 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container emptydir-injector ready: false, restart count 0 Jan 28 17:45:03.222: INFO: csi-mockplugin-attacher-0 started at 2023-01-28 17:40:56 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:45:03.222: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-zvdzg started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:03.222: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mkkk7 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:03.222: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:03.985: INFO: Latency metrics for node i-06d4799a5db582858 Jan 28 17:45:03.985: INFO: Logging node info for node i-0d3b431203a4a372c Jan 28 17:45:04.200: INFO: Node Info: &Node{ObjectMeta:{i-0d3b431203a4a372c b69212ba-b617-41e9-91e3-ee5929b727b2 34621 0 2023-01-28 17:22:12 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a io.kubernetes.storage.mock/node:some-mock-node kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d3b431203a4a372c kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0d3b431203a4a372c topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.33.52 csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-3106":"csi-mock-csi-mock-volumes-3106","csi-mock-csi-mock-volumes-5984":"i-0d3b431203a4a372c","ebs.csi.aws.com":"i-0d3b431203a4a372c"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {e2e.test Update v1 2023-01-28 17:39:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:45:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0d3b431203a4a372c,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:44:50 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:44:50 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:44:50 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:44:50 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.52,},NodeAddress{Type:ExternalIP,Address:13.229.100.227,},NodeAddress{Type:InternalDNS,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-229-100-227.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2fe0eb6b322558c1427d0b9ac3c84d,SystemUUID:ec2fe0eb-6b32-2558-c142-7d0b9ac3c84d,BootID:47c658c8-8593-40bf-a02e-622027452e1a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:45:04.200: INFO: Logging kubelet events for node i-0d3b431203a4a372c Jan 28 17:45:04.417: INFO: Logging pods the kubelet thinks is on node i-0d3b431203a4a372c Jan 28 17:45:04.640: INFO: csi-mockplugin-0 started at 2023-01-28 17:40:16 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:04.640: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:45:04.640: INFO: Container driver-registrar ready: true, restart count 0 Jan 28 17:45:04.640: INFO: Container mock ready: true, restart count 0 Jan 28 17:45:04.640: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-45pqn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:04.640: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:04.640: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-z6hs2 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:04.640: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:04.640: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-7bq4k started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:04.640: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:04.640: INFO: rs-n86z8 started at 2023-01-28 17:40:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:04.640: INFO: Container donothing ready: false, restart count 0 Jan 28 17:45:04.640: INFO: csi-mockplugin-attacher-0 started at 2023-01-28 17:40:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:04.640: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:45:04.640: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-kmzfw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:04.640: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:04.640: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-fzmt8 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:04.640: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:04.640: INFO: ebs-csi-node-j2pwn started at 2023-01-28 17:22:13 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:04.640: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:04.640: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:04.640: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:45:04.640: INFO: coredns-6dd78b7459-t8lf4 started at 2023-01-28 17:22:48 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:04.640: INFO: Container coredns ready: true, restart count 0 Jan 28 17:45:04.640: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-g7jvs started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:04.640: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:04.640: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-f8tsq started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:04.640: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:04.640: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6mbxw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:04.640: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:04.640: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mmn5l started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:04.641: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:04.641: INFO: cilium-cs2hk started at 2023-01-28 17:22:13 +0000 UTC (1+1 container statuses recorded) Jan 28 17:45:04.641: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:45:04.641: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:45:04.641: INFO: csi-mockplugin-0 started at 2023-01-28 17:41:21 +0000 UTC (0+4 container statuses recorded) Jan 28 17:45:04.641: INFO: Container busybox ready: true, restart count 0 Jan 28 17:45:04.641: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:45:04.641: INFO: Container driver-registrar ready: true, restart count 0 Jan 28 17:45:04.641: INFO: Container mock ready: true, restart count 0 Jan 28 17:45:04.641: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-hnjmr started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:04.641: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:05.406: INFO: Latency metrics for node i-0d3b431203a4a372c Jan 28 17:45:05.407: INFO: Logging node info for node i-0e10813507581b097 Jan 28 17:45:05.622: INFO: Node Info: &Node{ObjectMeta:{i-0e10813507581b097 8af9c758-a6f6-4a0d-b634-4d99ed71b929 32762 0 2023-01-28 17:22:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e10813507581b097 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0e10813507581b097 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.58 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e10813507581b097"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {e2e.test Update v1 2023-01-28 17:39:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:40:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0e10813507581b097,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:40:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.58,},NodeAddress{Type:ExternalIP,Address:54.254.187.204,},NodeAddress{Type:InternalDNS,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-187-204.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2922b3ab12fd264b764b33da3e62c2,SystemUUID:ec2922b3-ab12-fd26-4b76-4b33da3e62c2,BootID:07cb73db-252f-45aa-8a98-af3a42c0313b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:45:05.622: INFO: Logging kubelet events for node i-0e10813507581b097 Jan 28 17:45:05.839: INFO: Logging pods the kubelet thinks is on node i-0e10813507581b097 Jan 28 17:45:06.063: INFO: up-down-2-2j47t started at 2023-01-28 17:38:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container up-down-2 ready: true, restart count 0 Jan 28 17:45:06.063: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-2j7n9 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:06.063: INFO: pod-736ddca1-ca58-4462-8ac4-c2357b897d98 started at 2023-01-28 17:40:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:45:06.063: INFO: ebs-csi-node-k6rdn started at 2023-01-28 17:22:08 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:06.063: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:06.063: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:06.063: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:45:06.063: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-vnxp9 started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:06.063: INFO: csi-mockplugin-0 started at 2023-01-28 17:45:01 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:06.063: INFO: Container csi-provisioner ready: false, restart count 0 Jan 28 17:45:06.063: INFO: Container driver-registrar ready: false, restart count 0 Jan 28 17:45:06.063: INFO: Container mock ready: false, restart count 0 Jan 28 17:45:06.063: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-4hpf7 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:06.063: INFO: hostexec-i-0e10813507581b097-zj6lj started at 2023-01-28 17:40:20 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:45:06.063: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-x5gs8 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:06.063: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-9hkr5 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:06.063: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-22rht started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:06.063: INFO: coredns-6dd78b7459-qk7rr started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container coredns ready: true, restart count 0 Jan 28 17:45:06.063: INFO: test-ss-0 started at 2023-01-28 17:39:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container webserver ready: true, restart count 0 Jan 28 17:45:06.063: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-89jgx started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:06.063: INFO: up-down-3-bmxqn started at 2023-01-28 17:39:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container up-down-3 ready: false, restart count 0 Jan 28 17:45:06.063: INFO: coredns-autoscaler-5f76bf6c7b-nnmkq started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container autoscaler ready: true, restart count 0 Jan 28 17:45:06.063: INFO: csi-mockplugin-attacher-0 started at 2023-01-28 17:45:02 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container csi-attacher ready: false, restart count 0 Jan 28 17:45:06.063: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-rl6v2 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:06.063: INFO: pod-client started at 2023-01-28 17:40:19 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container pod-client ready: true, restart count 0 Jan 28 17:45:06.063: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mscfw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:06.063: INFO: cilium-mbfkc started at 2023-01-28 17:22:08 +0000 UTC (1+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:45:06.063: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:45:06.063: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-62lqj started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:06.063: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:06.794: INFO: Latency metrics for node i-0e10813507581b097 Jan 28 17:45:06.795: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "kubectl-1169" for this suite.
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-cli\]\sKubectl\sclient\sUpdate\sDemo\sshould\sscale\sa\sreplication\scontroller\s\s\[Conformance\]$'
test/e2e/framework/framework.go:652 Jan 28 17:49:57.825: Timed out after 300 seconds waiting for name=update-demo pods to reach valid state test/e2e/kubectl/kubectl.go:328from junit_17.xml
[BeforeEach] [sig-cli] Kubectl client test/e2e/framework/framework.go:187 �[1mSTEP�[0m: Creating a kubernetes client Jan 28 17:44:48.915: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename kubectl �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-cli] Kubectl client test/e2e/kubectl/kubectl.go:245 [BeforeEach] Update Demo test/e2e/kubectl/kubectl.go:297 [It] should scale a replication controller [Conformance] test/e2e/framework/framework.go:652 �[1mSTEP�[0m: creating a replication controller Jan 28 17:44:50.413: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 create -f -' Jan 28 17:44:51.500: INFO: stderr: "" Jan 28 17:44:51.500: INFO: stdout: "replicationcontroller/update-demo-nautilus created\n" �[1mSTEP�[0m: waiting for all containers in name=update-demo pods to come up. Jan 28 17:44:51.500: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:44:52.260: INFO: stderr: "" Jan 28 17:44:52.260: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:44:52.260: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:44:53.014: INFO: stderr: "" Jan 28 17:44:53.014: INFO: stdout: "" Jan 28 17:44:53.014: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:44:58.018: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:44:58.761: INFO: stderr: "" Jan 28 17:44:58.761: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:44:58.761: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:44:59.518: INFO: stderr: "" Jan 28 17:44:59.518: INFO: stdout: "" Jan 28 17:44:59.518: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:45:04.518: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:45:05.259: INFO: stderr: "" Jan 28 17:45:05.259: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:45:05.259: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:45:05.995: INFO: stderr: "" Jan 28 17:45:05.996: INFO: stdout: "" Jan 28 17:45:05.996: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:45:10.998: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:45:11.763: INFO: stderr: "" Jan 28 17:45:11.763: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:45:11.763: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:45:12.505: INFO: stderr: "" Jan 28 17:45:12.506: INFO: stdout: "" Jan 28 17:45:12.506: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:45:17.510: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:45:18.252: INFO: stderr: "" Jan 28 17:45:18.252: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:45:18.252: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:45:18.987: INFO: stderr: "" Jan 28 17:45:18.987: INFO: stdout: "" Jan 28 17:45:18.987: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:45:23.988: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:45:24.730: INFO: stderr: "" Jan 28 17:45:24.730: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:45:24.730: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:45:25.462: INFO: stderr: "" Jan 28 17:45:25.462: INFO: stdout: "" Jan 28 17:45:25.462: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:45:30.463: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:45:31.263: INFO: stderr: "" Jan 28 17:45:31.263: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:45:31.263: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:45:32.002: INFO: stderr: "" Jan 28 17:45:32.002: INFO: stdout: "" Jan 28 17:45:32.002: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:45:37.003: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:45:37.800: INFO: stderr: "" Jan 28 17:45:37.800: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:45:37.800: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:45:38.539: INFO: stderr: "" Jan 28 17:45:38.539: INFO: stdout: "" Jan 28 17:45:38.539: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:45:43.540: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:45:44.293: INFO: stderr: "" Jan 28 17:45:44.293: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:45:44.293: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:45:45.022: INFO: stderr: "" Jan 28 17:45:45.022: INFO: stdout: "" Jan 28 17:45:45.022: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:45:50.022: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:45:50.794: INFO: stderr: "" Jan 28 17:45:50.794: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:45:50.795: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:45:51.523: INFO: stderr: "" Jan 28 17:45:51.523: INFO: stdout: "" Jan 28 17:45:51.523: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:45:56.525: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:45:57.290: INFO: stderr: "" Jan 28 17:45:57.290: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:45:57.290: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:45:58.055: INFO: stderr: "" Jan 28 17:45:58.055: INFO: stdout: "" Jan 28 17:45:58.055: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:46:03.059: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:46:03.802: INFO: stderr: "" Jan 28 17:46:03.802: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:46:03.802: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:46:04.537: INFO: stderr: "" Jan 28 17:46:04.537: INFO: stdout: "" Jan 28 17:46:04.537: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:46:09.539: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:46:10.287: INFO: stderr: "" Jan 28 17:46:10.287: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:46:10.287: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:46:11.035: INFO: stderr: "" Jan 28 17:46:11.035: INFO: stdout: "" Jan 28 17:46:11.035: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:46:16.036: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:46:16.802: INFO: stderr: "" Jan 28 17:46:16.802: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:46:16.802: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:46:17.541: INFO: stderr: "" Jan 28 17:46:17.541: INFO: stdout: "" Jan 28 17:46:17.541: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:46:22.541: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:46:23.281: INFO: stderr: "" Jan 28 17:46:23.281: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:46:23.281: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:46:24.033: INFO: stderr: "" Jan 28 17:46:24.033: INFO: stdout: "" Jan 28 17:46:24.033: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:46:29.033: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:46:29.780: INFO: stderr: "" Jan 28 17:46:29.780: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:46:29.780: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:46:30.533: INFO: stderr: "" Jan 28 17:46:30.533: INFO: stdout: "" Jan 28 17:46:30.533: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:46:35.533: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:46:36.295: INFO: stderr: "" Jan 28 17:46:36.295: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:46:36.295: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:46:37.027: INFO: stderr: "" Jan 28 17:46:37.027: INFO: stdout: "" Jan 28 17:46:37.027: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:46:42.027: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:46:42.770: INFO: stderr: "" Jan 28 17:46:42.771: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:46:42.771: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:46:43.519: INFO: stderr: "" Jan 28 17:46:43.519: INFO: stdout: "" Jan 28 17:46:43.519: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:46:48.519: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:46:49.262: INFO: stderr: "" Jan 28 17:46:49.262: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:46:49.262: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:46:50.001: INFO: stderr: "" Jan 28 17:46:50.001: INFO: stdout: "" Jan 28 17:46:50.001: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:46:55.003: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:46:55.736: INFO: stderr: "" Jan 28 17:46:55.736: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:46:55.736: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:46:56.469: INFO: stderr: "" Jan 28 17:46:56.469: INFO: stdout: "" Jan 28 17:46:56.469: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:47:01.471: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:47:02.215: INFO: stderr: "" Jan 28 17:47:02.215: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:47:02.215: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:47:02.961: INFO: stderr: "" Jan 28 17:47:02.961: INFO: stdout: "" Jan 28 17:47:02.961: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:47:07.965: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:47:08.703: INFO: stderr: "" Jan 28 17:47:08.703: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:47:08.704: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:47:09.431: INFO: stderr: "" Jan 28 17:47:09.431: INFO: stdout: "" Jan 28 17:47:09.431: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:47:14.431: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:47:15.177: INFO: stderr: "" Jan 28 17:47:15.177: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:47:15.177: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:47:15.925: INFO: stderr: "" Jan 28 17:47:15.925: INFO: stdout: "" Jan 28 17:47:15.925: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:47:20.927: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:47:21.662: INFO: stderr: "" Jan 28 17:47:21.662: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:47:21.662: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:47:22.393: INFO: stderr: "" Jan 28 17:47:22.393: INFO: stdout: "" Jan 28 17:47:22.393: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:47:27.393: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:47:28.131: INFO: stderr: "" Jan 28 17:47:28.131: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:47:28.131: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:47:28.865: INFO: stderr: "" Jan 28 17:47:28.865: INFO: stdout: "" Jan 28 17:47:28.865: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:47:33.865: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:47:34.608: INFO: stderr: "" Jan 28 17:47:34.608: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:47:34.609: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:47:35.339: INFO: stderr: "" Jan 28 17:47:35.339: INFO: stdout: "" Jan 28 17:47:35.339: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:47:40.340: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:47:41.076: INFO: stderr: "" Jan 28 17:47:41.076: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:47:41.076: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:47:41.805: INFO: stderr: "" Jan 28 17:47:41.805: INFO: stdout: "" Jan 28 17:47:41.805: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:47:46.807: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:47:47.561: INFO: stderr: "" Jan 28 17:47:47.562: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:47:47.562: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:47:48.298: INFO: stderr: "" Jan 28 17:47:48.298: INFO: stdout: "" Jan 28 17:47:48.298: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:47:53.299: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:47:54.032: INFO: stderr: "" Jan 28 17:47:54.032: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:47:54.032: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:47:54.789: INFO: stderr: "" Jan 28 17:47:54.789: INFO: stdout: "" Jan 28 17:47:54.789: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:47:59.789: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:48:00.519: INFO: stderr: "" Jan 28 17:48:00.519: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:48:00.519: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:48:01.248: INFO: stderr: "" Jan 28 17:48:01.248: INFO: stdout: "" Jan 28 17:48:01.248: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:48:06.250: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:48:06.987: INFO: stderr: "" Jan 28 17:48:06.987: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:48:06.987: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:48:07.719: INFO: stderr: "" Jan 28 17:48:07.719: INFO: stdout: "" Jan 28 17:48:07.719: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:48:12.719: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:48:13.458: INFO: stderr: "" Jan 28 17:48:13.459: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:48:13.459: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:48:14.187: INFO: stderr: "" Jan 28 17:48:14.187: INFO: stdout: "" Jan 28 17:48:14.187: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:48:19.187: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:48:20.137: INFO: stderr: "" Jan 28 17:48:20.137: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:48:20.137: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:48:20.888: INFO: stderr: "" Jan 28 17:48:20.888: INFO: stdout: "" Jan 28 17:48:20.888: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:48:25.888: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:48:26.847: INFO: stderr: "" Jan 28 17:48:26.847: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:48:26.847: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:48:27.589: INFO: stderr: "" Jan 28 17:48:27.589: INFO: stdout: "" Jan 28 17:48:27.589: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:48:32.591: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:48:33.340: INFO: stderr: "" Jan 28 17:48:33.340: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:48:33.340: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:48:34.075: INFO: stderr: "" Jan 28 17:48:34.075: INFO: stdout: "" Jan 28 17:48:34.075: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:48:39.075: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:48:39.825: INFO: stderr: "" Jan 28 17:48:39.825: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:48:39.825: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:48:40.569: INFO: stderr: "" Jan 28 17:48:40.569: INFO: stdout: "" Jan 28 17:48:40.569: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:48:45.570: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:48:46.312: INFO: stderr: "" Jan 28 17:48:46.312: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:48:46.312: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:48:47.058: INFO: stderr: "" Jan 28 17:48:47.058: INFO: stdout: "" Jan 28 17:48:47.058: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:48:52.058: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:48:52.836: INFO: stderr: "" Jan 28 17:48:52.837: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:48:52.837: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:48:53.593: INFO: stderr: "" Jan 28 17:48:53.593: INFO: stdout: "" Jan 28 17:48:53.593: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:48:58.594: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:48:59.575: INFO: stderr: "" Jan 28 17:48:59.575: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:48:59.575: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:49:00.346: INFO: stderr: "" Jan 28 17:49:00.346: INFO: stdout: "" Jan 28 17:49:00.346: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:49:05.346: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:49:06.110: INFO: stderr: "" Jan 28 17:49:06.111: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:49:06.111: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:49:06.879: INFO: stderr: "" Jan 28 17:49:06.879: INFO: stdout: "" Jan 28 17:49:06.879: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:49:11.883: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:49:12.642: INFO: stderr: "" Jan 28 17:49:12.642: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:49:12.642: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:49:13.382: INFO: stderr: "" Jan 28 17:49:13.382: INFO: stdout: "" Jan 28 17:49:13.382: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:49:18.383: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:49:19.134: INFO: stderr: "" Jan 28 17:49:19.134: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:49:19.134: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:49:19.871: INFO: stderr: "" Jan 28 17:49:19.871: INFO: stdout: "" Jan 28 17:49:19.871: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:49:24.871: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:49:25.821: INFO: stderr: "" Jan 28 17:49:25.821: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:49:25.821: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:49:26.561: INFO: stderr: "" Jan 28 17:49:26.561: INFO: stdout: "" Jan 28 17:49:26.561: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:49:31.561: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:49:32.323: INFO: stderr: "" Jan 28 17:49:32.323: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:49:32.323: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:49:33.085: INFO: stderr: "" Jan 28 17:49:33.085: INFO: stdout: "" Jan 28 17:49:33.085: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:49:38.086: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:49:39.043: INFO: stderr: "" Jan 28 17:49:39.043: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:49:39.043: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:49:39.793: INFO: stderr: "" Jan 28 17:49:39.793: INFO: stdout: "" Jan 28 17:49:39.793: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:49:44.794: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:49:45.554: INFO: stderr: "" Jan 28 17:49:45.554: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:49:45.554: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:49:46.317: INFO: stderr: "" Jan 28 17:49:46.317: INFO: stdout: "" Jan 28 17:49:46.317: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:49:51.319: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo' Jan 28 17:49:52.071: INFO: stderr: "" Jan 28 17:49:52.071: INFO: stdout: "update-demo-nautilus-vl9xg update-demo-nautilus-zqp4x " Jan 28 17:49:52.071: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods update-demo-nautilus-vl9xg -o template --template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if (and (eq .name "update-demo") (exists . "state" "running"))}}true{{end}}{{end}}{{end}}' Jan 28 17:49:52.824: INFO: stderr: "" Jan 28 17:49:52.824: INFO: stdout: "" Jan 28 17:49:52.824: INFO: update-demo-nautilus-vl9xg is created but not running Jan 28 17:49:57.825: FAIL: Timed out after 300 seconds waiting for name=update-demo pods to reach valid state Full Stack Trace k8s.io/kubernetes/test/e2e/kubectl.glob..func1.6.3() test/e2e/kubectl/kubectl.go:328 +0x205 k8s.io/kubernetes/test/e2e.RunE2ETests(0x25761d7?) test/e2e/e2e.go:130 +0x6bb k8s.io/kubernetes/test/e2e.TestE2E(0x24e52d9?) test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc000500b60, 0x741f9a8) /usr/local/go/src/testing/testing.go:1446 +0x10b created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1493 +0x35f �[1mSTEP�[0m: using delete to clean up resources Jan 28 17:49:57.825: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 delete --grace-period=0 --force -f -' Jan 28 17:49:58.778: INFO: stderr: "warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.\n" Jan 28 17:49:58.778: INFO: stdout: "replicationcontroller \"update-demo-nautilus\" force deleted\n" Jan 28 17:49:58.778: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get rc,svc -l name=update-demo --no-headers' Jan 28 17:49:59.726: INFO: stderr: "No resources found in kubectl-8855 namespace.\n" Jan 28 17:49:59.726: INFO: stdout: "" Jan 28 17:49:59.726: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-8855 get pods -l name=update-demo -o go-template={{ range .items }}{{ if not .metadata.deletionTimestamp }}{{ .metadata.name }}{{ "\n" }}{{ end }}{{ end }}' Jan 28 17:50:00.485: INFO: stderr: "" Jan 28 17:50:00.485: INFO: stdout: "" [AfterEach] [sig-cli] Kubectl client test/e2e/framework/framework.go:188 �[1mSTEP�[0m: Collecting events from namespace "kubectl-8855". �[1mSTEP�[0m: Found 29 events. Jan 28 17:50:00.700: INFO: At 2023-01-28 17:44:51 +0000 UTC - event for update-demo-nautilus: {replication-controller } SuccessfulCreate: Created pod: update-demo-nautilus-zqp4x Jan 28 17:50:00.700: INFO: At 2023-01-28 17:44:51 +0000 UTC - event for update-demo-nautilus: {replication-controller } SuccessfulCreate: Created pod: update-demo-nautilus-vl9xg Jan 28 17:50:00.700: INFO: At 2023-01-28 17:44:51 +0000 UTC - event for update-demo-nautilus-vl9xg: {default-scheduler } Scheduled: Successfully assigned kubectl-8855/update-demo-nautilus-vl9xg to i-03342b7c6ebddc1c6 Jan 28 17:50:00.700: INFO: At 2023-01-28 17:44:51 +0000 UTC - event for update-demo-nautilus-vl9xg: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f1ca7dbfe445509881fd5e0db74bbaa8e1b30d766d3d44f5122aaa7ea7c00389": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:44:51 +0000 UTC - event for update-demo-nautilus-zqp4x: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "681e0a5cafbdc754822a121d47537c7843d22e3af0a508369d221e4aba0006b1": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:44:51 +0000 UTC - event for update-demo-nautilus-zqp4x: {default-scheduler } Scheduled: Successfully assigned kubectl-8855/update-demo-nautilus-zqp4x to i-06d4799a5db582858 Jan 28 17:50:00.700: INFO: At 2023-01-28 17:45:04 +0000 UTC - event for update-demo-nautilus-vl9xg: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fd5c7464fb022ae0412ff6b972d7b0aa42cea533d112078fc47df461a5e40d00": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:45:04 +0000 UTC - event for update-demo-nautilus-zqp4x: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2e785e6e1f2abf96d1ca19bfeb8a933f7e52a7038c78570df1be74b28b16000a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:45:17 +0000 UTC - event for update-demo-nautilus-vl9xg: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b83fa3adbad92fc9a6cae5ce3b5ad65151cf0862c8664047c21e8462e7fc6339": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:45:19 +0000 UTC - event for update-demo-nautilus-zqp4x: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6537fb14825925a4469c4776c609322447c84bae07dacb10e1385bca5e72e4fc": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:45:30 +0000 UTC - event for update-demo-nautilus-vl9xg: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "dd2a3cf4839320cf151e922d98285f39d6fd608a9c4d1305a85c39376c9c8b28": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:45:32 +0000 UTC - event for update-demo-nautilus-zqp4x: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b53227b2b20d3e42130cc2fa11d44fa38d48714332d3817c41060b4f95b2bb34": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:45:46 +0000 UTC - event for update-demo-nautilus-vl9xg: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "05e2a15895466b538c9af7e48a0a0512a8ec4e4f0c0cd6211fb799217f444541": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:45:46 +0000 UTC - event for update-demo-nautilus-zqp4x: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "20c90a51a5d7bc81dd3393448857e2f33cde79ede4e6a522c3162a4a377ffd74": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:45:57 +0000 UTC - event for update-demo-nautilus-zqp4x: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3ceb2de902617cbdaa1c74e9ddd951d286d0c5db3c594b49f375870e87a8ada7": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:46:00 +0000 UTC - event for update-demo-nautilus-vl9xg: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "33a015c5d195b868c3bc1b9223165c8babc44bf2e1c334ddcaa200196ec406b7": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:46:09 +0000 UTC - event for update-demo-nautilus-zqp4x: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "721b3c039e44c9e2f34c5ba733fb139ed0a4d0f33f4ed8af0fa1f5c24678116f": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:46:13 +0000 UTC - event for update-demo-nautilus-vl9xg: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3bb76cceda6ff9c0b953af16c4cc5599e20dc60ee1fd651abf9a92b4abe2f6ea": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:46:22 +0000 UTC - event for update-demo-nautilus-zqp4x: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "de94e71a0992bfb7628b9c62a6cc860e6d3feda18b1410a34d19cb336365badb": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:46:25 +0000 UTC - event for update-demo-nautilus-vl9xg: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4451eb6dfaae26365afb6c0387ee1453a3a9384fd2f0278d9fe49a8942de81f2": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:46:37 +0000 UTC - event for update-demo-nautilus-vl9xg: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a60b05bcdc18203cf4971278130f7423913eceebcf202968ae012001cff8978c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:46:37 +0000 UTC - event for update-demo-nautilus-zqp4x: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e293f663987aba8fa44d736abb07ce01f2f7a42f5b29a18521be9ee4a3b6833c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:46:49 +0000 UTC - event for update-demo-nautilus-vl9xg: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a5b03c843c7e5d963ab62f7c245523d35dad2a252bb78426cbf79a47a6da17a3": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:46:50 +0000 UTC - event for update-demo-nautilus-zqp4x: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "98c870a4c73c6f7f2c7f22ac8efa03eab3ff8588387ebd2d15b1309d3608c9aa": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:00.700: INFO: At 2023-01-28 17:47:53 +0000 UTC - event for update-demo-nautilus-zqp4x: {kubelet i-06d4799a5db582858} Pulling: Pulling image "k8s.gcr.io/e2e-test-images/nautilus:1.5" Jan 28 17:50:00.700: INFO: At 2023-01-28 17:47:58 +0000 UTC - event for update-demo-nautilus-zqp4x: {kubelet i-06d4799a5db582858} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/nautilus:1.5" in 4.618913052s Jan 28 17:50:00.700: INFO: At 2023-01-28 17:47:58 +0000 UTC - event for update-demo-nautilus-zqp4x: {kubelet i-06d4799a5db582858} Created: Created container update-demo Jan 28 17:50:00.700: INFO: At 2023-01-28 17:47:58 +0000 UTC - event for update-demo-nautilus-zqp4x: {kubelet i-06d4799a5db582858} Started: Started container update-demo Jan 28 17:50:00.700: INFO: At 2023-01-28 17:49:58 +0000 UTC - event for update-demo-nautilus-zqp4x: {kubelet i-06d4799a5db582858} Killing: Stopping container update-demo Jan 28 17:50:00.914: INFO: POD NODE PHASE GRACE CONDITIONS Jan 28 17:50:00.914: INFO: Jan 28 17:50:01.129: INFO: Logging node info for node i-03342b7c6ebddc1c6 Jan 28 17:50:01.344: INFO: Node Info: &Node{ObjectMeta:{i-03342b7c6ebddc1c6 2b79525a-8646-4141-9334-8d4a7a592265 36017 0 2023-01-28 17:22:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03342b7c6ebddc1c6 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-03342b7c6ebddc1c6 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.39.74 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03342b7c6ebddc1c6"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {e2e.test Update v1 2023-01-28 17:39:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:47:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-03342b7c6ebddc1c6,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.74,},NodeAddress{Type:ExternalIP,Address:52.221.218.125,},NodeAddress{Type:InternalDNS,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-218-125.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bb29c4b9b2fc4087668bfcd056585,SystemUUID:ec2bb29c-4b9b-2fc4-0876-68bfcd056585,BootID:042ee65a-a182-44df-9f74-80b116af3725,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:50:01.344: INFO: Logging kubelet events for node i-03342b7c6ebddc1c6 Jan 28 17:50:01.560: INFO: Logging pods the kubelet thinks is on node i-03342b7c6ebddc1c6 Jan 28 17:50:01.799: INFO: test-rollover-controller-nvj8x started at 2023-01-28 17:48:00 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container httpd ready: false, restart count 0 Jan 28 17:50:01.799: INFO: pod-8b1ca53e-c2c1-4d2c-8354-ebf444c76264 started at 2023-01-28 17:47:07 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:50:01.799: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6b7pn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:01.799: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mnncj started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:01.799: INFO: failed-jobs-history-limit-27915468-zhmr9 started at 2023-01-28 17:48:00 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container c ready: false, restart count 0 Jan 28 17:50:01.799: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-kcssq started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:01.799: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-57xvw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:01.799: INFO: ebs-csi-node-xxvv5 started at 2023-01-28 17:22:12 +0000 UTC (0+3 container statuses recorded) Jan 28 17:50:01.799: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:50:01.799: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:50:01.799: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:50:01.799: INFO: affinity-nodeport-2vxfn started at 2023-01-28 17:47:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container affinity-nodeport ready: false, restart count 0 Jan 28 17:50:01.799: INFO: echo-sourceip started at 2023-01-28 17:48:29 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container agnhost-container ready: false, restart count 0 Jan 28 17:50:01.799: INFO: e2e-6bwsq-2vnvv started at 2023-01-28 17:45:45 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container c ready: false, restart count 0 Jan 28 17:50:01.799: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-jqzfn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:01.799: INFO: failed-jobs-history-limit-27915470-7flcb started at 2023-01-28 17:50:00 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container c ready: false, restart count 0 Jan 28 17:50:01.799: INFO: cilium-wlwtf started at 2023-01-28 17:22:12 +0000 UTC (1+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:50:01.799: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:50:01.799: INFO: hostexec-i-03342b7c6ebddc1c6-s4cmv started at 2023-01-28 17:44:49 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:50:01.799: INFO: slave started at 2023-01-28 17:46:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container cntr ready: false, restart count 0 Jan 28 17:50:01.799: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6vfpn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:01.799: INFO: pod-projected-configmaps-232ce7a1-e69f-4c43-88ec-9422d5087ff4 started at 2023-01-28 17:47:29 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container agnhost-container ready: false, restart count 0 Jan 28 17:50:01.799: INFO: httpd started at 2023-01-28 17:47:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container httpd ready: false, restart count 0 Jan 28 17:50:01.799: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-9jldw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:01.799: INFO: pod1 started at 2023-01-28 17:45:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:50:01.799: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mkb65 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:01.799: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-fgg6g started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:01.799: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-ch9br started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:01.799: INFO: master started at 2023-01-28 17:44:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container cntr ready: true, restart count 0 Jan 28 17:50:01.799: INFO: pod-0 started at 2023-01-28 17:47:09 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:01.799: INFO: Container donothing ready: false, restart count 0 Jan 28 17:50:02.757: INFO: Latency metrics for node i-03342b7c6ebddc1c6 Jan 28 17:50:02.757: INFO: Logging node info for node i-0550ecdc8b2275df1 Jan 28 17:50:02.972: INFO: Node Info: &Node{ObjectMeta:{i-0550ecdc8b2275df1 e2f1fe4c-911f-4da0-9481-51f08c3e654e 37029 0 2023-01-28 17:20:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0550ecdc8b2275df1 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.60.182 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0550ecdc8b2275df1"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:20:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-01-28 17:20:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-01-28 17:21:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-28 17:22:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0550ecdc8b2275df1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:21:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.182,},NodeAddress{Type:ExternalIP,Address:52.221.229.242,},NodeAddress{Type:InternalDNS,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-229-242.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec22a890f5274dd8ae5c60b32c44aae9,SystemUUID:ec22a890-f527-4dd8-ae5c-60b32c44aae9,BootID:83ea7fa2-7c97-4062-a52f-a856b154a02c,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:231502799,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64@sha256:6db2761074bae8385aa601d4b45ab065893cfdc258d8b2593fc56698c44dbee5 k8s.gcr.io/kube-apiserver-amd64:v1.24.10 registry.k8s.io/kube-apiserver-amd64:v1.24.10],SizeBytes:131733971,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64@sha256:8867df47c332bae8dfd061b4d4230cd0febcc5a8a427ecb2e8f70b33ffc6d732 k8s.gcr.io/kube-controller-manager-amd64:v1.24.10 registry.k8s.io/kube-controller-manager-amd64:v1.24.10],SizeBytes:121342265,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64@sha256:a589f10748fa227ea6ebf0730126f3f192cbdbb2c7f96a9d4685cabb86eb1cbc k8s.gcr.io/kube-scheduler-amd64:v1.24.10 registry.k8s.io/kube-scheduler-amd64:v1.24.10],SizeBytes:52751160,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.2],SizeBytes:43830714,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.2],SizeBytes:42821730,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:26802430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:245e81c2ed53b5508a6f687f3ab0c724b9ab31e6a5fa5e6d2f814981ab6ce55f registry.k8s.io/provider-aws/cloud-controller-manager:v1.24.3],SizeBytes:18282737,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.2],SizeBytes:4965809,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:50:02.972: INFO: Logging kubelet events for node i-0550ecdc8b2275df1 Jan 28 17:50:03.188: INFO: Logging pods the kubelet thinks is on node i-0550ecdc8b2275df1 Jan 28 17:50:03.411: INFO: etcd-manager-events-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:03.411: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:50:03.411: INFO: etcd-manager-main-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:03.411: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:50:03.411: INFO: kube-apiserver-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+2 container statuses recorded) Jan 28 17:50:03.411: INFO: Container healthcheck ready: true, restart count 0 Jan 28 17:50:03.411: INFO: Container kube-apiserver ready: true, restart count 0 Jan 28 17:50:03.411: INFO: kube-controller-manager-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:03.411: INFO: Container kube-controller-manager ready: true, restart count 3 Jan 28 17:50:03.411: INFO: kops-controller-gll72 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:03.411: INFO: Container kops-controller ready: true, restart count 0 Jan 28 17:50:03.411: INFO: cilium-4trt7 started at 2023-01-28 17:21:03 +0000 UTC (1+1 container statuses recorded) Jan 28 17:50:03.411: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:50:03.411: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:50:03.411: INFO: aws-cloud-controller-manager-w6p6f started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:03.411: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Jan 28 17:50:03.411: INFO: cilium-operator-8d8478c74-2vsdk started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:03.411: INFO: Container cilium-operator ready: true, restart count 0 Jan 28 17:50:03.411: INFO: ebs-csi-controller-779575c67d-kx2cv started at 2023-01-28 17:21:03 +0000 UTC (0+5 container statuses recorded) Jan 28 17:50:03.411: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:50:03.411: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:50:03.411: INFO: Container csi-resizer ready: true, restart count 0 Jan 28 17:50:03.411: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:50:03.411: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:50:03.411: INFO: kube-scheduler-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:03.411: INFO: Container kube-scheduler ready: true, restart count 0 Jan 28 17:50:03.411: INFO: ebs-csi-node-stxgw started at 2023-01-28 17:21:03 +0000 UTC (0+3 container statuses recorded) Jan 28 17:50:03.411: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:50:03.411: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:50:03.411: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:50:03.411: INFO: dns-controller-7d4957564c-nd8s8 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:03.411: INFO: Container dns-controller ready: true, restart count 0 Jan 28 17:50:04.100: INFO: Latency metrics for node i-0550ecdc8b2275df1 Jan 28 17:50:04.100: INFO: Logging node info for node i-06d4799a5db582858 Jan 28 17:50:04.314: INFO: Node Info: &Node{ObjectMeta:{i-06d4799a5db582858 aacf1bf6-d2bd-4f61-aad7-83531f5d9ba1 36980 0 2023-01-28 17:22:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-06d4799a5db582858 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-06d4799a5db582858 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.139 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06d4799a5db582858"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {e2e.test Update v1 2023-01-28 17:39:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:40:14 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 17:40:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-06d4799a5db582858,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.139,},NodeAddress{Type:ExternalIP,Address:18.143.137.61,},NodeAddress{Type:InternalDNS,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-143-137-61.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6f00e5a83cc422a6226665122e9d,SystemUUID:ec2e6f00-e5a8-3cc4-22a6-226665122e9d,BootID:3bfeb6d1-b807-4cfa-a0a7-954aef9e3823,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:49642095,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0870183487a8091e2 kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0870183487a8091e2,DevicePath:,},},Config:nil,},} Jan 28 17:50:04.314: INFO: Logging kubelet events for node i-06d4799a5db582858 Jan 28 17:50:04.531: INFO: Logging pods the kubelet thinks is on node i-06d4799a5db582858 Jan 28 17:50:04.759: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-jq2xw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:04.759: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-7dx5z started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:04.759: INFO: pod-270e9e95-df4d-4169-9c6c-9329994f97a6 started at 2023-01-28 17:43:18 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:50:04.759: INFO: test-rs-2cpll started at 2023-01-28 17:45:37 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container httpd ready: false, restart count 0 Jan 28 17:50:04.759: INFO: pod-1 started at 2023-01-28 17:47:09 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container donothing ready: false, restart count 0 Jan 28 17:50:04.759: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-r9lbh started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:04.759: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-slm8r started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:04.759: INFO: pod2 started at 2023-01-28 17:45:40 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container agnhost-container ready: false, restart count 0 Jan 28 17:50:04.759: INFO: e2e-6bwsq-8fx8x started at 2023-01-28 17:45:45 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container c ready: false, restart count 0 Jan 28 17:50:04.759: INFO: failed-jobs-history-limit-27915467-n9hrr started at 2023-01-28 17:47:00 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container c ready: false, restart count 0 Jan 28 17:50:04.759: INFO: hostexec-i-06d4799a5db582858-4b8ml started at 2023-01-28 17:47:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:50:04.759: INFO: failed-jobs-history-limit-27915469-f2qfh started at 2023-01-28 17:49:00 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container c ready: false, restart count 0 Jan 28 17:50:04.759: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-t9lsx started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:04.759: INFO: terminate-cmd-rpof3f8a92e2-bf59-4664-8855-611405125631 started at 2023-01-28 17:47:26 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container terminate-cmd-rpof ready: false, restart count 0 Jan 28 17:50:04.759: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mkkk7 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:04.759: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-zvdzg started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:04.759: INFO: pod-12357688-a276-4eb3-96ce-6a76a486ef13 started at 2023-01-28 17:47:18 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:50:04.759: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-dgjbr started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:04.759: INFO: e2e-6bwsq-qxq2h started at 2023-01-28 17:47:56 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container c ready: false, restart count 0 Jan 28 17:50:04.759: INFO: cilium-8cb5h started at 2023-01-28 17:22:14 +0000 UTC (1+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:50:04.759: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:50:04.759: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-bdtqt started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:04.759: INFO: affinity-nodeport-9d4b5 started at 2023-01-28 17:47:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container affinity-nodeport ready: true, restart count 0 Jan 28 17:50:04.759: INFO: local-injector started at 2023-01-28 17:47:20 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container local-injector ready: false, restart count 0 Jan 28 17:50:04.759: INFO: pod-configmaps-ec72283f-7902-4782-87b5-eaa55eeb6a4c started at 2023-01-28 17:47:49 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container env-test ready: false, restart count 0 Jan 28 17:50:04.759: INFO: e2e-6bwsq-7dz2j started at 2023-01-28 17:46:22 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container c ready: false, restart count 0 Jan 28 17:50:04.759: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-lw565 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:04.759: INFO: ephemeral-containers-target-pod started at 2023-01-28 17:47:14 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:04.759: INFO: Container test-container-1 ready: false, restart count 0 Jan 28 17:50:04.759: INFO: ebs-csi-node-ttfmk started at 2023-01-28 17:22:14 +0000 UTC (0+3 container statuses recorded) Jan 28 17:50:04.759: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:50:04.759: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:50:04.759: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:50:05.552: INFO: Latency metrics for node i-06d4799a5db582858 Jan 28 17:50:05.552: INFO: Logging node info for node i-0d3b431203a4a372c Jan 28 17:50:05.766: INFO: Node Info: &Node{ObjectMeta:{i-0d3b431203a4a372c b69212ba-b617-41e9-91e3-ee5929b727b2 34827 0 2023-01-28 17:22:12 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a io.kubernetes.storage.mock/node:some-mock-node kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d3b431203a4a372c kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0d3b431203a4a372c topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.33.52 csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-3106":"csi-mock-csi-mock-volumes-3106","ebs.csi.aws.com":"i-0d3b431203a4a372c"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {e2e.test Update v1 2023-01-28 17:39:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:45:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0d3b431203a4a372c,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:45:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:45:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:45:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:45:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.52,},NodeAddress{Type:ExternalIP,Address:13.229.100.227,},NodeAddress{Type:InternalDNS,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-229-100-227.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2fe0eb6b322558c1427d0b9ac3c84d,SystemUUID:ec2fe0eb-6b32-2558-c142-7d0b9ac3c84d,BootID:47c658c8-8593-40bf-a02e-622027452e1a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:50:05.767: INFO: Logging kubelet events for node i-0d3b431203a4a372c Jan 28 17:50:05.983: INFO: Logging pods the kubelet thinks is on node i-0d3b431203a4a372c Jan 28 17:50:06.207: INFO: csi-mockplugin-resizer-0 started at 2023-01-28 17:47:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:06.207: INFO: Container csi-resizer ready: false, restart count 0 Jan 28 17:50:06.207: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-kmzfw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:06.207: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:06.207: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-fzmt8 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:06.208: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:06.208: INFO: ebs-csi-node-j2pwn started at 2023-01-28 17:22:13 +0000 UTC (0+3 container statuses recorded) Jan 28 17:50:06.208: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:50:06.208: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:50:06.208: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:50:06.208: INFO: coredns-6dd78b7459-t8lf4 started at 2023-01-28 17:22:48 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:06.208: INFO: Container coredns ready: true, restart count 0 Jan 28 17:50:06.208: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-g7jvs started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:06.208: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:06.208: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-f8tsq started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:06.208: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:06.208: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6mbxw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:06.208: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:06.208: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mmn5l started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:06.208: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:06.208: INFO: cilium-cs2hk started at 2023-01-28 17:22:13 +0000 UTC (1+1 container statuses recorded) Jan 28 17:50:06.208: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:50:06.208: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:50:06.208: INFO: csi-mockplugin-0 started at 2023-01-28 17:41:21 +0000 UTC (0+4 container statuses recorded) Jan 28 17:50:06.208: INFO: Container busybox ready: true, restart count 0 Jan 28 17:50:06.208: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:50:06.208: INFO: Container driver-registrar ready: true, restart count 0 Jan 28 17:50:06.208: INFO: Container mock ready: true, restart count 0 Jan 28 17:50:06.208: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-hnjmr started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:06.208: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:06.208: INFO: csi-mockplugin-0 started at 2023-01-28 17:47:23 +0000 UTC (0+3 container statuses recorded) Jan 28 17:50:06.208: INFO: Container csi-provisioner ready: false, restart count 0 Jan 28 17:50:06.208: INFO: Container driver-registrar ready: false, restart count 0 Jan 28 17:50:06.208: INFO: Container mock ready: false, restart count 0 Jan 28 17:50:06.208: INFO: pvc-volume-tester-8szsf started at 2023-01-28 17:45:15 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:06.208: INFO: Container volume-tester ready: false, restart count 0 Jan 28 17:50:06.208: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-45pqn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:06.208: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:06.208: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-z6hs2 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:06.208: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:06.208: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-7bq4k started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:06.208: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:06.971: INFO: Latency metrics for node i-0d3b431203a4a372c Jan 28 17:50:06.971: INFO: Logging node info for node i-0e10813507581b097 Jan 28 17:50:07.185: INFO: Node Info: &Node{ObjectMeta:{i-0e10813507581b097 8af9c758-a6f6-4a0d-b634-4d99ed71b929 34939 0 2023-01-28 17:22:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e10813507581b097 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0e10813507581b097 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.58 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e10813507581b097"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {e2e.test Update v1 2023-01-28 17:39:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:40:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0e10813507581b097,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:45:27 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:45:27 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:45:27 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:45:27 +0000 UTC,LastTransitionTime:2023-01-28 17:22:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.58,},NodeAddress{Type:ExternalIP,Address:54.254.187.204,},NodeAddress{Type:InternalDNS,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-187-204.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2922b3ab12fd264b764b33da3e62c2,SystemUUID:ec2922b3-ab12-fd26-4b76-4b33da3e62c2,BootID:07cb73db-252f-45aa-8a98-af3a42c0313b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:50:07.185: INFO: Logging kubelet events for node i-0e10813507581b097 Jan 28 17:50:07.401: INFO: Logging pods the kubelet thinks is on node i-0e10813507581b097 Jan 28 17:50:07.636: INFO: cilium-mbfkc started at 2023-01-28 17:22:08 +0000 UTC (1+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:50:07.636: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:50:07.636: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-62lqj started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:50:07.636: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mscfw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:07.636: INFO: boom-server started at 2023-01-28 17:48:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container boom-server ready: false, restart count 0 Jan 28 17:50:07.636: INFO: ebs-csi-node-k6rdn started at 2023-01-28 17:22:08 +0000 UTC (0+3 container statuses recorded) Jan 28 17:50:07.636: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:50:07.636: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:50:07.636: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:50:07.636: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-vnxp9 started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:07.636: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-2j7n9 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:07.636: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-4hpf7 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:07.636: INFO: hostexec-i-0e10813507581b097-7x65b started at 2023-01-28 17:45:59 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:50:07.636: INFO: csi-mockplugin-0 started at 2023-01-28 17:45:01 +0000 UTC (0+3 container statuses recorded) Jan 28 17:50:07.636: INFO: Container csi-provisioner ready: false, restart count 0 Jan 28 17:50:07.636: INFO: Container driver-registrar ready: false, restart count 0 Jan 28 17:50:07.636: INFO: Container mock ready: false, restart count 0 Jan 28 17:50:07.636: INFO: affinity-nodeport-2mdr9 started at 2023-01-28 17:47:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container affinity-nodeport ready: false, restart count 0 Jan 28 17:50:07.636: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-x5gs8 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:07.636: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-9hkr5 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:07.636: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-22rht started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:07.636: INFO: coredns-6dd78b7459-qk7rr started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container coredns ready: true, restart count 0 Jan 28 17:50:07.636: INFO: pod-0e370f03-c117-4110-99af-41ac7db026f8 started at 2023-01-28 17:46:04 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:50:07.636: INFO: coredns-autoscaler-5f76bf6c7b-nnmkq started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container autoscaler ready: true, restart count 0 Jan 28 17:50:07.636: INFO: csi-mockplugin-attacher-0 started at 2023-01-28 17:45:02 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:50:07.636: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-89jgx started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:07.636: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-rl6v2 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:07.636: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:08.434: INFO: Latency metrics for node i-0e10813507581b097 Jan 28 17:50:08.434: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "kubectl-8855" for this suite.
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-network\]\sConntrack\sshould\sbe\sable\sto\spreserve\sUDP\straffic\swhen\sserver\spod\scycles\sfor\sa\sClusterIP\sservice$'
test/e2e/network/conntrack.go:208 Jan 28 17:45:23.018: Unexpected error: <*errors.errorString | 0xc000209d30>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred test/e2e/framework/pods.go:107from junit_14.xml
{"msg":"PASSED [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand should resize volume when PVC is edited while pod is using it","total":-1,"completed":21,"skipped":217,"failed":0} [BeforeEach] [sig-network] Conntrack test/e2e/framework/framework.go:187 �[1mSTEP�[0m: Creating a kubernetes client Jan 28 17:40:17.112: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename conntrack �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-network] Conntrack test/e2e/network/conntrack.go:98 [It] should be able to preserve UDP traffic when server pod cycles for a ClusterIP service test/e2e/network/conntrack.go:208 �[1mSTEP�[0m: creating a UDP service svc-udp with type=ClusterIP in conntrack-4869 �[1mSTEP�[0m: creating a client pod for probing the service svc-udp Jan 28 17:40:19.500: INFO: The status of Pod pod-client is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:21.716: INFO: The status of Pod pod-client is Running (Ready = true) Jan 28 17:40:22.148: INFO: Pod client logs: Sat Jan 28 17:40:21 UTC 2023 Sat Jan 28 17:40:21 UTC 2023 Try: 1 �[1mSTEP�[0m: creating a backend pod pod-server-1 for the service svc-udp Jan 28 17:40:22.583: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:24.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:26.802: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:28.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:30.799: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:32.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:34.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:36.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:38.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:40.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:42.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:44.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:46.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:48.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:50.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:52.802: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:54.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:56.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:58.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:00.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:02.802: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:04.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:06.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:08.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:10.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:12.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:14.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:16.799: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:18.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:20.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:22.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:24.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:26.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:28.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:30.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:32.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:34.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:36.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:38.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:40.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:42.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:44.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:46.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:48.799: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:50.799: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:52.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:54.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:56.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:58.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:00.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:02.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:04.804: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:06.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:08.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:10.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:12.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:14.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:16.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:18.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:20.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:22.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:24.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:26.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:28.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:30.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:32.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:34.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:36.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:38.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:40.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:42.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:44.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:46.803: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:48.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:50.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:52.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:54.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:56.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:58.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:00.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:02.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:04.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:06.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:08.799: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:10.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:12.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:14.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:16.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:18.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:20.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:22.799: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:24.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:26.799: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:28.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:30.806: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:32.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:34.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:36.799: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:38.799: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:40.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:42.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:44.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:46.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:48.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:50.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:52.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:54.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:56.803: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:58.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:00.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:02.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:04.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:06.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:08.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:10.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:12.802: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:14.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:16.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:18.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:20.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:22.799: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:24.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:26.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:28.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:30.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:32.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:34.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:36.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:38.802: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:40.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:42.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:44.799: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:46.808: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:48.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:50.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:52.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:54.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:56.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:58.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:00.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:02.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:04.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:06.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:08.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:10.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:12.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:14.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:16.800: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:18.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:20.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:22.801: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:23.018: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:23.018: FAIL: Unexpected error: <*errors.errorString | 0xc000209d30>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/framework.(*PodClient).CreateSync(0xc0030eaf00, 0xc00130d230?) test/e2e/framework/pods.go:107 +0x94 k8s.io/kubernetes/test/e2e/network.glob..func1.4() test/e2e/network/conntrack.go:242 +0x645 k8s.io/kubernetes/test/e2e.RunE2ETests(0x25761d7?) test/e2e/e2e.go:130 +0x6bb k8s.io/kubernetes/test/e2e.TestE2E(0x0?) test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc0001029c0, 0x741f9a8) /usr/local/go/src/testing/testing.go:1446 +0x10b created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1493 +0x35f [AfterEach] [sig-network] Conntrack test/e2e/framework/framework.go:188 �[1mSTEP�[0m: Collecting events from namespace "conntrack-4869". �[1mSTEP�[0m: Found 15 events. Jan 28 17:45:23.235: INFO: At 2023-01-28 17:40:19 +0000 UTC - event for pod-client: {default-scheduler } Scheduled: Successfully assigned conntrack-4869/pod-client to i-0e10813507581b097 Jan 28 17:45:23.236: INFO: At 2023-01-28 17:40:21 +0000 UTC - event for pod-client: {kubelet i-0e10813507581b097} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:23.236: INFO: At 2023-01-28 17:40:21 +0000 UTC - event for pod-client: {kubelet i-0e10813507581b097} Created: Created container pod-client Jan 28 17:45:23.236: INFO: At 2023-01-28 17:40:21 +0000 UTC - event for pod-client: {kubelet i-0e10813507581b097} Started: Started container pod-client Jan 28 17:45:23.236: INFO: At 2023-01-28 17:40:22 +0000 UTC - event for pod-server-1: {default-scheduler } Scheduled: Successfully assigned conntrack-4869/pod-server-1 to i-06d4799a5db582858 Jan 28 17:45:23.236: INFO: At 2023-01-28 17:40:22 +0000 UTC - event for pod-server-1: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c7ef9980f281d3f50ec4e1059a464aa83386fd2b1c74b4cfdaf78c225fcb6c7d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:23.236: INFO: At 2023-01-28 17:40:34 +0000 UTC - event for pod-server-1: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "67b06245618b55ef590fec3670f2fd137f89a04450fbdc1a8e681375ae60e33f": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:23.236: INFO: At 2023-01-28 17:40:45 +0000 UTC - event for pod-server-1: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2eca77d300395c167d59add9032e2f858c66d9b96000e776b16f777e45f9aec9": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:23.236: INFO: At 2023-01-28 17:40:56 +0000 UTC - event for pod-server-1: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5447eec2a22ff79efde269c8be76c841e13350e6bbe016800388d17dd207b1f9": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:23.236: INFO: At 2023-01-28 17:41:09 +0000 UTC - event for pod-server-1: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2ae178b341074a8f0d53e9a8086e65b869478f248dbcd2dc7b79e635b56ae7f1": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:23.236: INFO: At 2023-01-28 17:41:24 +0000 UTC - event for pod-server-1: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b6eff1c460b83b18b3255890a7713d35264f41d830a5f94e9b281cb90926ea65": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:23.236: INFO: At 2023-01-28 17:41:38 +0000 UTC - event for pod-server-1: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0a712cc52d5f62f85398c1f1fddc15f88b3993c198f21485a5e71c8919d47433": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:23.236: INFO: At 2023-01-28 17:41:51 +0000 UTC - event for pod-server-1: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "bd4ee9784518cf355131210e5520128c94e181714e1709917afdb76c2e09c3bd": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:23.236: INFO: At 2023-01-28 17:42:05 +0000 UTC - event for pod-server-1: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6d5e177be4038d3e07f94392faa3cc739f3e4cc75a46b99977939cc2fb34689d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:23.236: INFO: At 2023-01-28 17:42:16 +0000 UTC - event for pod-server-1: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "62f46ea62417a3094fbae5f56c1a96ad16a350f0923e112fa355ca12b6bfcc1c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:23.452: INFO: POD NODE PHASE GRACE CONDITIONS Jan 28 17:45:23.452: INFO: pod-client i-0e10813507581b097 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:40:19 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:40:21 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:40:21 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:40:19 +0000 UTC }] Jan 28 17:45:23.452: INFO: pod-server-1 i-06d4799a5db582858 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:40:22 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:40:22 +0000 UTC ContainersNotReady containers with unready status: [agnhost-container]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:40:22 +0000 UTC ContainersNotReady containers with unready status: [agnhost-container]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:40:22 +0000 UTC }] Jan 28 17:45:23.452: INFO: Jan 28 17:45:23.888: INFO: Unable to fetch conntrack-4869/pod-server-1/agnhost-container logs: the server rejected our request for an unknown reason (get pods pod-server-1) Jan 28 17:45:24.107: INFO: Logging node info for node i-03342b7c6ebddc1c6 Jan 28 17:45:24.324: INFO: Node Info: &Node{ObjectMeta:{i-03342b7c6ebddc1c6 2b79525a-8646-4141-9334-8d4a7a592265 34811 0 2023-01-28 17:22:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03342b7c6ebddc1c6 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-03342b7c6ebddc1c6 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.39.74 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03342b7c6ebddc1c6"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:37:15 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 17:37:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {e2e.test Update v1 2023-01-28 17:39:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-03342b7c6ebddc1c6,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:45:19 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:45:19 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:45:19 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:45:19 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.74,},NodeAddress{Type:ExternalIP,Address:52.221.218.125,},NodeAddress{Type:InternalDNS,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-218-125.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bb29c4b9b2fc4087668bfcd056585,SystemUUID:ec2bb29c-4b9b-2fc4-0876-68bfcd056585,BootID:042ee65a-a182-44df-9f74-80b116af3725,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-02c58416f403003e0],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02c58416f403003e0,DevicePath:,},},Config:nil,},} Jan 28 17:45:24.324: INFO: Logging kubelet events for node i-03342b7c6ebddc1c6 Jan 28 17:45:24.543: INFO: Logging pods the kubelet thinks is on node i-03342b7c6ebddc1c6 Jan 28 17:45:24.772: INFO: adopt-release-56c4c started at 2023-01-28 17:42:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.772: INFO: Container c ready: false, restart count 0 Jan 28 17:45:24.772: INFO: metadata-volume-38632bfd-b241-439f-8c2d-9f517e3aaee4 started at 2023-01-28 17:45:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.772: INFO: Container client-container ready: false, restart count 0 Jan 28 17:45:24.772: INFO: rs-4ztpt started at 2023-01-28 17:40:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.772: INFO: Container donothing ready: true, restart count 0 Jan 28 17:45:24.772: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6b7pn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.772: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:24.772: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mnncj started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.772: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:24.772: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-kcssq started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.772: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:24.772: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-57xvw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.772: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:24.772: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-jqzfn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.772: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:24.772: INFO: ebs-csi-node-xxvv5 started at 2023-01-28 17:22:12 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:24.772: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:24.772: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:24.772: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:45:24.772: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6vfpn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.772: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:24.772: INFO: cilium-wlwtf started at 2023-01-28 17:22:12 +0000 UTC (1+1 container statuses recorded) Jan 28 17:45:24.772: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:45:24.772: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:45:24.772: INFO: hostexec-i-03342b7c6ebddc1c6-s4cmv started at 2023-01-28 17:44:49 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.773: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:45:24.773: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-9jldw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.773: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:24.773: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mkb65 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.773: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:24.773: INFO: pod-d024b2fd-b2af-4707-9028-d6c200999384 started at 2023-01-28 17:44:57 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.773: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:45:24.773: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-fgg6g started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.773: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:24.773: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-ch9br started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.773: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:24.773: INFO: master started at 2023-01-28 17:44:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.773: INFO: Container cntr ready: false, restart count 0 Jan 28 17:45:24.773: INFO: test-deployment-9dxdt-688c4d6789-4hbzm started at 2023-01-28 17:45:10 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.773: INFO: Container httpd ready: false, restart count 0 Jan 28 17:45:24.773: INFO: pod-23984187-216b-4e72-b531-729449648f82 started at 2023-01-28 17:43:00 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.773: INFO: Container test-container ready: false, restart count 0 Jan 28 17:45:24.773: INFO: update-demo-nautilus-vl9xg started at 2023-01-28 17:44:51 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.773: INFO: Container update-demo ready: false, restart count 0 Jan 28 17:45:24.773: INFO: ss-1 started at 2023-01-28 17:39:40 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.773: INFO: Container webserver ready: false, restart count 0 Jan 28 17:45:24.773: INFO: test-ss-1 started at 2023-01-28 17:40:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:24.773: INFO: Container webserver ready: false, restart count 0 Jan 28 17:45:25.729: INFO: Latency metrics for node i-03342b7c6ebddc1c6 Jan 28 17:45:25.729: INFO: Logging node info for node i-0550ecdc8b2275df1 Jan 28 17:45:25.945: INFO: Node Info: &Node{ObjectMeta:{i-0550ecdc8b2275df1 e2f1fe4c-911f-4da0-9481-51f08c3e654e 33948 0 2023-01-28 17:20:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0550ecdc8b2275df1 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.60.182 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0550ecdc8b2275df1"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:20:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-01-28 17:20:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-01-28 17:21:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-28 17:22:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0550ecdc8b2275df1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:21:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.182,},NodeAddress{Type:ExternalIP,Address:52.221.229.242,},NodeAddress{Type:InternalDNS,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-229-242.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec22a890f5274dd8ae5c60b32c44aae9,SystemUUID:ec22a890-f527-4dd8-ae5c-60b32c44aae9,BootID:83ea7fa2-7c97-4062-a52f-a856b154a02c,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:231502799,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64@sha256:6db2761074bae8385aa601d4b45ab065893cfdc258d8b2593fc56698c44dbee5 k8s.gcr.io/kube-apiserver-amd64:v1.24.10 registry.k8s.io/kube-apiserver-amd64:v1.24.10],SizeBytes:131733971,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64@sha256:8867df47c332bae8dfd061b4d4230cd0febcc5a8a427ecb2e8f70b33ffc6d732 k8s.gcr.io/kube-controller-manager-amd64:v1.24.10 registry.k8s.io/kube-controller-manager-amd64:v1.24.10],SizeBytes:121342265,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64@sha256:a589f10748fa227ea6ebf0730126f3f192cbdbb2c7f96a9d4685cabb86eb1cbc k8s.gcr.io/kube-scheduler-amd64:v1.24.10 registry.k8s.io/kube-scheduler-amd64:v1.24.10],SizeBytes:52751160,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.2],SizeBytes:43830714,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.2],SizeBytes:42821730,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:26802430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:245e81c2ed53b5508a6f687f3ab0c724b9ab31e6a5fa5e6d2f814981ab6ce55f registry.k8s.io/provider-aws/cloud-controller-manager:v1.24.3],SizeBytes:18282737,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.2],SizeBytes:4965809,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:45:25.946: INFO: Logging kubelet events for node i-0550ecdc8b2275df1 Jan 28 17:45:26.164: INFO: Logging pods the kubelet thinks is on node i-0550ecdc8b2275df1 Jan 28 17:45:26.388: INFO: kube-scheduler-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:26.388: INFO: Container kube-scheduler ready: true, restart count 0 Jan 28 17:45:26.388: INFO: ebs-csi-node-stxgw started at 2023-01-28 17:21:03 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:26.388: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:26.388: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:26.388: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:45:26.388: INFO: dns-controller-7d4957564c-nd8s8 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:26.388: INFO: Container dns-controller ready: true, restart count 0 Jan 28 17:45:26.388: INFO: aws-cloud-controller-manager-w6p6f started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:26.388: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Jan 28 17:45:26.388: INFO: cilium-operator-8d8478c74-2vsdk started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:26.388: INFO: Container cilium-operator ready: true, restart count 0 Jan 28 17:45:26.388: INFO: etcd-manager-events-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:26.388: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:45:26.388: INFO: etcd-manager-main-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:26.388: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:45:26.388: INFO: kube-apiserver-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+2 container statuses recorded) Jan 28 17:45:26.388: INFO: Container healthcheck ready: true, restart count 0 Jan 28 17:45:26.388: INFO: Container kube-apiserver ready: true, restart count 0 Jan 28 17:45:26.388: INFO: kube-controller-manager-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:26.388: INFO: Container kube-controller-manager ready: true, restart count 3 Jan 28 17:45:26.388: INFO: kops-controller-gll72 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:26.388: INFO: Container kops-controller ready: true, restart count 0 Jan 28 17:45:26.388: INFO: cilium-4trt7 started at 2023-01-28 17:21:03 +0000 UTC (1+1 container statuses recorded) Jan 28 17:45:26.388: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:45:26.388: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:45:26.388: INFO: ebs-csi-controller-779575c67d-kx2cv started at 2023-01-28 17:21:03 +0000 UTC (0+5 container statuses recorded) Jan 28 17:45:26.388: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:45:26.389: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:45:26.389: INFO: Container csi-resizer ready: true, restart count 0 Jan 28 17:45:26.389: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:26.389: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:27.081: INFO: Latency metrics for node i-0550ecdc8b2275df1 Jan 28 17:45:27.081: INFO: Logging node info for node i-06d4799a5db582858 Jan 28 17:45:27.298: INFO: Node Info: &Node{ObjectMeta:{i-06d4799a5db582858 aacf1bf6-d2bd-4f61-aad7-83531f5d9ba1 33004 0 2023-01-28 17:22:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-06d4799a5db582858 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-06d4799a5db582858 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.139 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06d4799a5db582858"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {e2e.test Update v1 2023-01-28 17:39:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:40:14 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 17:40:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-06d4799a5db582858,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.139,},NodeAddress{Type:ExternalIP,Address:18.143.137.61,},NodeAddress{Type:InternalDNS,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-143-137-61.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6f00e5a83cc422a6226665122e9d,SystemUUID:ec2e6f00-e5a8-3cc4-22a6-226665122e9d,BootID:3bfeb6d1-b807-4cfa-a0a7-954aef9e3823,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc kubernetes.io/csi/ebs.csi.aws.com^vol-0dff02acc75d59bc3],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0dff02acc75d59bc3,DevicePath:,},},Config:nil,},} Jan 28 17:45:27.298: INFO: Logging kubelet events for node i-06d4799a5db582858 Jan 28 17:45:27.519: INFO: Logging pods the kubelet thinks is on node i-06d4799a5db582858 Jan 28 17:45:27.746: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-bdtqt started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:27.746: INFO: pod-server-1 started at 2023-01-28 17:40:22 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container agnhost-container ready: false, restart count 0 Jan 28 17:45:27.746: INFO: csi-mockplugin-0 started at 2023-01-28 17:40:55 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:27.746: INFO: Container csi-provisioner ready: false, restart count 0 Jan 28 17:45:27.746: INFO: Container driver-registrar ready: false, restart count 0 Jan 28 17:45:27.746: INFO: Container mock ready: false, restart count 0 Jan 28 17:45:27.746: INFO: ebs-csi-node-ttfmk started at 2023-01-28 17:22:14 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:27.746: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:27.746: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:27.746: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:45:27.746: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-lw565 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:27.746: INFO: busybox-702a2763-bdb2-44b8-ae2d-94e4cd0026af started at 2023-01-28 17:42:05 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container busybox ready: false, restart count 0 Jan 28 17:45:27.746: INFO: adopt-release-g5t8f started at 2023-01-28 17:42:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container c ready: false, restart count 0 Jan 28 17:45:27.746: INFO: rs-tbjch started at 2023-01-28 17:40:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container donothing ready: true, restart count 0 Jan 28 17:45:27.746: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-jq2xw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:27.746: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-7dx5z started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:27.746: INFO: pod-270e9e95-df4d-4169-9c6c-9329994f97a6 started at 2023-01-28 17:43:18 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:45:27.746: INFO: dns-test-bffe5683-c9dc-4647-9125-fa445279f23f started at 2023-01-28 17:45:16 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:27.746: INFO: Container jessie-querier ready: false, restart count 0 Jan 28 17:45:27.746: INFO: Container querier ready: false, restart count 0 Jan 28 17:45:27.746: INFO: Container webserver ready: false, restart count 0 Jan 28 17:45:27.746: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-t9lsx started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:27.746: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-r9lbh started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:27.746: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-slm8r started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:27.746: INFO: csi-mockplugin-attacher-0 started at 2023-01-28 17:40:56 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:45:27.746: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-zvdzg started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:27.746: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mkkk7 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:27.746: INFO: emptydir-injector started at 2023-01-28 17:40:31 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container emptydir-injector ready: false, restart count 0 Jan 28 17:45:27.746: INFO: pod-configmaps-7cd33bfb-d4f5-467e-946d-cdf07018d8cf started at 2023-01-28 17:43:01 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container env-test ready: false, restart count 0 Jan 28 17:45:27.746: INFO: httpd started at 2023-01-28 17:44:35 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container httpd ready: false, restart count 0 Jan 28 17:45:27.746: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-dgjbr started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:27.746: INFO: cilium-8cb5h started at 2023-01-28 17:22:14 +0000 UTC (1+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:45:27.746: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:45:27.746: INFO: ss-0 started at 2023-01-28 17:40:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container webserver ready: false, restart count 0 Jan 28 17:45:27.746: INFO: update-demo-nautilus-zqp4x started at 2023-01-28 17:44:51 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:27.746: INFO: Container update-demo ready: false, restart count 0 Jan 28 17:45:28.506: INFO: Latency metrics for node i-06d4799a5db582858 Jan 28 17:45:28.506: INFO: Logging node info for node i-0d3b431203a4a372c Jan 28 17:45:28.723: INFO: Node Info: &Node{ObjectMeta:{i-0d3b431203a4a372c b69212ba-b617-41e9-91e3-ee5929b727b2 34827 0 2023-01-28 17:22:12 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a io.kubernetes.storage.mock/node:some-mock-node kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d3b431203a4a372c kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0d3b431203a4a372c topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.33.52 csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-3106":"csi-mock-csi-mock-volumes-3106","ebs.csi.aws.com":"i-0d3b431203a4a372c"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {e2e.test Update v1 2023-01-28 17:39:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:45:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0d3b431203a4a372c,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:45:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:45:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:45:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:45:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.52,},NodeAddress{Type:ExternalIP,Address:13.229.100.227,},NodeAddress{Type:InternalDNS,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-229-100-227.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2fe0eb6b322558c1427d0b9ac3c84d,SystemUUID:ec2fe0eb-6b32-2558-c142-7d0b9ac3c84d,BootID:47c658c8-8593-40bf-a02e-622027452e1a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:45:28.723: INFO: Logging kubelet events for node i-0d3b431203a4a372c Jan 28 17:45:28.942: INFO: Logging pods the kubelet thinks is on node i-0d3b431203a4a372c Jan 28 17:45:29.171: INFO: pvc-volume-tester-8szsf started at 2023-01-28 17:45:15 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:29.171: INFO: Container volume-tester ready: false, restart count 0 Jan 28 17:45:29.171: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-45pqn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:29.171: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:29.171: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-z6hs2 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:29.171: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:29.171: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-7bq4k started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:29.171: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:29.171: INFO: rs-n86z8 started at 2023-01-28 17:40:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:29.171: INFO: Container donothing ready: true, restart count 0 Jan 28 17:45:29.171: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-kmzfw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:29.171: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:29.171: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-fzmt8 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:29.171: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:29.171: INFO: ebs-csi-node-j2pwn started at 2023-01-28 17:22:13 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:29.171: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:29.171: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:29.171: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:45:29.171: INFO: coredns-6dd78b7459-t8lf4 started at 2023-01-28 17:22:48 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:29.171: INFO: Container coredns ready: true, restart count 0 Jan 28 17:45:29.171: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-g7jvs started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:29.171: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:29.171: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-f8tsq started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:29.171: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:29.171: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6mbxw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:29.171: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:29.171: INFO: cilium-cs2hk started at 2023-01-28 17:22:13 +0000 UTC (1+1 container statuses recorded) Jan 28 17:45:29.171: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:45:29.171: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:45:29.171: INFO: csi-mockplugin-0 started at 2023-01-28 17:41:21 +0000 UTC (0+4 container statuses recorded) Jan 28 17:45:29.171: INFO: Container busybox ready: true, restart count 0 Jan 28 17:45:29.171: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:45:29.171: INFO: Container driver-registrar ready: true, restart count 0 Jan 28 17:45:29.171: INFO: Container mock ready: true, restart count 0 Jan 28 17:45:29.171: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-hnjmr started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:29.171: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:29.171: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mmn5l started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:29.171: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:29.959: INFO: Latency metrics for node i-0d3b431203a4a372c Jan 28 17:45:29.959: INFO: Logging node info for node i-0e10813507581b097 Jan 28 17:45:30.176: INFO: Node Info: &Node{ObjectMeta:{i-0e10813507581b097 8af9c758-a6f6-4a0d-b634-4d99ed71b929 34939 0 2023-01-28 17:22:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e10813507581b097 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0e10813507581b097 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.58 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e10813507581b097"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {e2e.test Update v1 2023-01-28 17:39:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:40:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0e10813507581b097,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:45:27 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:45:27 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:45:27 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:45:27 +0000 UTC,LastTransitionTime:2023-01-28 17:22:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.58,},NodeAddress{Type:ExternalIP,Address:54.254.187.204,},NodeAddress{Type:InternalDNS,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-187-204.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2922b3ab12fd264b764b33da3e62c2,SystemUUID:ec2922b3-ab12-fd26-4b76-4b33da3e62c2,BootID:07cb73db-252f-45aa-8a98-af3a42c0313b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:45:30.176: INFO: Logging kubelet events for node i-0e10813507581b097 Jan 28 17:45:30.400: INFO: Logging pods the kubelet thinks is on node i-0e10813507581b097 Jan 28 17:45:30.625: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-x5gs8 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.625: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:30.625: INFO: coredns-6dd78b7459-qk7rr started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container coredns ready: true, restart count 0 Jan 28 17:45:30.626: INFO: test-ss-0 started at 2023-01-28 17:39:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container webserver ready: true, restart count 0 Jan 28 17:45:30.626: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-9hkr5 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:30.626: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-22rht started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:30.626: INFO: coredns-autoscaler-5f76bf6c7b-nnmkq started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container autoscaler ready: true, restart count 0 Jan 28 17:45:30.626: INFO: csi-mockplugin-attacher-0 started at 2023-01-28 17:45:02 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container csi-attacher ready: false, restart count 0 Jan 28 17:45:30.626: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-89jgx started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:30.626: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-rl6v2 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:30.626: INFO: cilium-mbfkc started at 2023-01-28 17:22:08 +0000 UTC (1+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:45:30.626: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:45:30.626: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-62lqj started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:30.626: INFO: pod-client started at 2023-01-28 17:40:19 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container pod-client ready: true, restart count 0 Jan 28 17:45:30.626: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mscfw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:30.626: INFO: ebs-csi-node-k6rdn started at 2023-01-28 17:22:08 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:30.626: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:30.626: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:30.626: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:45:30.626: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-vnxp9 started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:30.626: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-2j7n9 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:30.626: INFO: pod-736ddca1-ca58-4462-8ac4-c2357b897d98 started at 2023-01-28 17:40:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:45:30.626: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-4hpf7 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:30.626: INFO: hostexec-i-0e10813507581b097-zj6lj started at 2023-01-28 17:40:20 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:30.626: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:45:30.626: INFO: csi-mockplugin-0 started at 2023-01-28 17:45:01 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:30.626: INFO: Container csi-provisioner ready: false, restart count 0 Jan 28 17:45:30.626: INFO: Container driver-registrar ready: false, restart count 0 Jan 28 17:45:30.626: INFO: Container mock ready: false, restart count 0 Jan 28 17:45:31.352: INFO: Latency metrics for node i-0e10813507581b097 Jan 28 17:45:31.352: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "conntrack-4869" for this suite.
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-network\]\sConntrack\sshould\sbe\sable\sto\spreserve\sUDP\straffic\swhen\sserver\spod\scycles\sfor\sa\sNodePort\sservice$'
test/e2e/network/conntrack.go:132 Jan 28 17:44:36.509: Unexpected error: <*errors.errorString | 0xc00021c1e0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred test/e2e/framework/pods.go:107from junit_17.xml
[BeforeEach] [sig-network] Conntrack test/e2e/framework/framework.go:187 �[1mSTEP�[0m: Creating a kubernetes client Jan 28 17:39:05.468: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename conntrack �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-network] Conntrack test/e2e/network/conntrack.go:98 [It] should be able to preserve UDP traffic when server pod cycles for a NodePort service test/e2e/network/conntrack.go:132 �[1mSTEP�[0m: creating a UDP service svc-udp with type=NodePort in conntrack-3826 �[1mSTEP�[0m: creating a client pod for probing the service svc-udp Jan 28 17:39:07.837: INFO: The status of Pod pod-client is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:10.053: INFO: The status of Pod pod-client is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:12.052: INFO: The status of Pod pod-client is Running (Ready = true) Jan 28 17:39:12.496: INFO: Pod client logs: Sat Jan 28 17:39:10 UTC 2023 Sat Jan 28 17:39:10 UTC 2023 Try: 1 �[1mSTEP�[0m: creating a backend pod pod-server-1 for the service svc-udp Jan 28 17:39:12.925: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:15.140: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:17.166: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:19.139: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:21.144: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:23.140: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:25.140: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:27.139: INFO: The status of Pod pod-server-1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:29.139: INFO: The status of Pod pod-server-1 is Running (Ready = true) �[1mSTEP�[0m: waiting up to 3m0s for service svc-udp in namespace conntrack-3826 to expose endpoints map[pod-server-1:[80]] Jan 28 17:39:30.208: INFO: successfully validated that service svc-udp in namespace conntrack-3826 exposes endpoints map[pod-server-1:[80]] �[1mSTEP�[0m: checking client pod connected to the backend 1 on Node IP 172.20.54.58 �[1mSTEP�[0m: creating a second backend pod pod-server-2 for the service svc-udp Jan 28 17:39:36.079: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:38.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:40.296: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:42.292: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:44.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:46.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:48.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:50.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:52.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:54.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:56.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:58.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:00.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:02.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:04.297: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:06.295: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:08.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:10.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:12.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:14.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:16.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:18.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:20.298: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:22.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:24.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:26.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:28.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:30.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:32.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:34.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:36.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:38.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:40.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:42.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:44.292: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:46.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:48.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:50.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:52.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:54.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:56.296: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:58.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:00.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:02.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:04.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:06.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:08.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:10.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:12.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:14.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:16.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:18.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:20.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:22.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:24.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:26.296: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:28.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:30.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:32.295: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:34.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:36.295: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:38.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:40.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:42.295: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:44.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:46.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:48.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:50.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:52.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:54.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:56.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:58.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:00.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:02.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:04.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:06.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:08.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:10.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:12.295: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:14.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:16.299: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:18.295: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:20.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:22.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:24.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:26.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:28.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:30.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:32.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:34.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:36.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:38.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:40.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:42.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:44.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:46.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:48.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:50.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:52.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:54.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:56.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:58.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:00.295: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:02.295: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:04.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:06.295: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:08.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:10.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:12.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:14.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:16.295: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:18.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:20.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:22.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:24.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:26.295: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:28.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:30.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:32.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:34.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:36.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:38.295: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:40.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:42.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:44.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:46.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:48.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:50.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:52.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:54.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:56.295: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:43:58.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:00.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:02.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:04.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:06.295: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:08.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:10.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:12.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:14.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:16.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:18.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:20.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:22.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:24.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:26.295: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:28.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:30.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:32.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:34.293: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:36.294: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:36.508: INFO: The status of Pod pod-server-2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:36.508: FAIL: Unexpected error: <*errors.errorString | 0xc00021c1e0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/framework.(*PodClient).CreateSync(0xc003e050c8, 0xc000dd3068?) test/e2e/framework/pods.go:107 +0x94 k8s.io/kubernetes/test/e2e/network.glob..func1.3() test/e2e/network/conntrack.go:189 +0xa1f k8s.io/kubernetes/test/e2e.RunE2ETests(0x25761d7?) test/e2e/e2e.go:130 +0x6bb k8s.io/kubernetes/test/e2e.TestE2E(0x24e52d9?) test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc000500b60, 0x741f9a8) /usr/local/go/src/testing/testing.go:1446 +0x10b created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1493 +0x35f [AfterEach] [sig-network] Conntrack test/e2e/framework/framework.go:188 �[1mSTEP�[0m: Collecting events from namespace "conntrack-3826". �[1mSTEP�[0m: Found 19 events. Jan 28 17:44:36.723: INFO: At 2023-01-28 17:39:07 +0000 UTC - event for pod-client: {default-scheduler } Scheduled: Successfully assigned conntrack-3826/pod-client to i-0d3b431203a4a372c Jan 28 17:44:36.723: INFO: At 2023-01-28 17:39:10 +0000 UTC - event for pod-client: {kubelet i-0d3b431203a4a372c} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:44:36.723: INFO: At 2023-01-28 17:39:10 +0000 UTC - event for pod-client: {kubelet i-0d3b431203a4a372c} Created: Created container pod-client Jan 28 17:44:36.723: INFO: At 2023-01-28 17:39:10 +0000 UTC - event for pod-client: {kubelet i-0d3b431203a4a372c} Started: Started container pod-client Jan 28 17:44:36.723: INFO: At 2023-01-28 17:39:12 +0000 UTC - event for pod-server-1: {default-scheduler } Scheduled: Successfully assigned conntrack-3826/pod-server-1 to i-0e10813507581b097 Jan 28 17:44:36.723: INFO: At 2023-01-28 17:39:16 +0000 UTC - event for pod-server-1: {kubelet i-0e10813507581b097} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:44:36.723: INFO: At 2023-01-28 17:39:16 +0000 UTC - event for pod-server-1: {kubelet i-0e10813507581b097} Created: Created container agnhost-container Jan 28 17:44:36.723: INFO: At 2023-01-28 17:39:17 +0000 UTC - event for pod-server-1: {kubelet i-0e10813507581b097} Started: Started container agnhost-container Jan 28 17:44:36.723: INFO: At 2023-01-28 17:39:35 +0000 UTC - event for pod-server-2: {default-scheduler } Scheduled: Successfully assigned conntrack-3826/pod-server-2 to i-0e10813507581b097 Jan 28 17:44:36.723: INFO: At 2023-01-28 17:39:36 +0000 UTC - event for pod-server-2: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "976a08bd85cc7b5563abf47c3e42185c137a966f89dd01abe1bbe14014e3ff4a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:36.723: INFO: At 2023-01-28 17:39:49 +0000 UTC - event for pod-server-2: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "287e5335feb624d519aea87c8e4a08c44c26bbedc89a6fe1e84a8085242d2d2a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:36.723: INFO: At 2023-01-28 17:40:04 +0000 UTC - event for pod-server-2: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1cf38af53e7e9130f4c76dcb2b413ab0e32294b927056933598536fa7087d9de": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:36.723: INFO: At 2023-01-28 17:40:16 +0000 UTC - event for pod-server-2: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4bbb7a02343171724af94bc4f724143043736b0c2d8c48deaab069139a05c6d1": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:36.723: INFO: At 2023-01-28 17:40:31 +0000 UTC - event for pod-server-2: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7fe78f1579568940d3c8a4b631684d53574d2fa14b1620a96d64431c6281e88a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:36.723: INFO: At 2023-01-28 17:40:46 +0000 UTC - event for pod-server-2: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fa0ead5240c17e1b5a96f82d9bbecc0d27067a65fe91dd4a97c8f3b99c3ab40f": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:36.723: INFO: At 2023-01-28 17:40:58 +0000 UTC - event for pod-server-2: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6f6a77212335beca140790b99a483734cd0c7da9dcc1d2c247afdebeff3b41a1": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:36.723: INFO: At 2023-01-28 17:41:11 +0000 UTC - event for pod-server-2: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f7b1e32a6e5e3b9f15d4ac94246c4cf4825c92da20690eb7c7fc7f43ace9738f": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:36.723: INFO: At 2023-01-28 17:41:26 +0000 UTC - event for pod-server-2: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cf1521a99265e1b36dde50d5d59aa5346579d60b90fb5b4dbce7f8d93ec39ba8": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:36.723: INFO: At 2023-01-28 17:41:41 +0000 UTC - event for pod-server-2: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "eb7d82944df4383e0c2cd48dee6dce6b349ca30cc387f5e9f3d1094998c654be": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:44:36.938: INFO: POD NODE PHASE GRACE CONDITIONS Jan 28 17:44:36.938: INFO: pod-client i-0d3b431203a4a372c Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:07 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:10 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:10 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:07 +0000 UTC }] Jan 28 17:44:36.938: INFO: pod-server-1 i-0e10813507581b097 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:12 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:18 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:18 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:12 +0000 UTC }] Jan 28 17:44:36.938: INFO: pod-server-2 i-0e10813507581b097 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:35 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:35 +0000 UTC ContainersNotReady containers with unready status: [agnhost-container]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:35 +0000 UTC ContainersNotReady containers with unready status: [agnhost-container]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:35 +0000 UTC }] Jan 28 17:44:36.938: INFO: Jan 28 17:44:37.590: INFO: Unable to fetch conntrack-3826/pod-server-2/agnhost-container logs: the server rejected our request for an unknown reason (get pods pod-server-2) Jan 28 17:44:37.806: INFO: Logging node info for node i-03342b7c6ebddc1c6 Jan 28 17:44:38.020: INFO: Node Info: &Node{ObjectMeta:{i-03342b7c6ebddc1c6 2b79525a-8646-4141-9334-8d4a7a592265 32812 0 2023-01-28 17:22:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03342b7c6ebddc1c6 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-03342b7c6ebddc1c6 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.39.74 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03342b7c6ebddc1c6"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:37:15 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 17:37:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {e2e.test Update v1 2023-01-28 17:39:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-03342b7c6ebddc1c6,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:13 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:13 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:13 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:40:13 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.74,},NodeAddress{Type:ExternalIP,Address:52.221.218.125,},NodeAddress{Type:InternalDNS,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-218-125.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bb29c4b9b2fc4087668bfcd056585,SystemUUID:ec2bb29c-4b9b-2fc4-0876-68bfcd056585,BootID:042ee65a-a182-44df-9f74-80b116af3725,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-02c58416f403003e0],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02c58416f403003e0,DevicePath:,},},Config:nil,},} Jan 28 17:44:38.020: INFO: Logging kubelet events for node i-03342b7c6ebddc1c6 Jan 28 17:44:38.236: INFO: Logging pods the kubelet thinks is on node i-03342b7c6ebddc1c6 Jan 28 17:44:38.672: INFO: test-ss-1 started at 2023-01-28 17:40:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container webserver ready: false, restart count 0 Jan 28 17:44:38.672: INFO: pod-23984187-216b-4e72-b531-729449648f82 started at 2023-01-28 17:43:00 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container test-container ready: false, restart count 0 Jan 28 17:44:38.672: INFO: hostexec-i-03342b7c6ebddc1c6-pcl5f started at 2023-01-28 17:39:19 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:44:38.672: INFO: ss-1 started at 2023-01-28 17:39:40 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container webserver ready: false, restart count 0 Jan 28 17:44:38.672: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mnncj started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:38.672: INFO: adopt-release-56c4c started at 2023-01-28 17:42:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container c ready: false, restart count 0 Jan 28 17:44:38.672: INFO: rs-4ztpt started at 2023-01-28 17:40:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container donothing ready: true, restart count 0 Jan 28 17:44:38.672: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6b7pn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:38.672: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-57xvw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:44:38.672: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-kcssq started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:38.672: INFO: up-down-2-rw86m started at 2023-01-28 17:38:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container up-down-2 ready: true, restart count 0 Jan 28 17:44:38.672: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-jqzfn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:38.672: INFO: pod-secrets-49e03bcd-e201-4e96-8caa-cef1cb071d5d started at 2023-01-28 17:40:18 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container secret-volume-test ready: false, restart count 0 Jan 28 17:44:38.672: INFO: ebs-csi-node-xxvv5 started at 2023-01-28 17:22:12 +0000 UTC (0+3 container statuses recorded) Jan 28 17:44:38.672: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:44:38.672: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:44:38.672: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:44:38.672: INFO: up-down-3-rcvxq started at 2023-01-28 17:39:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container up-down-3 ready: true, restart count 0 Jan 28 17:44:38.672: INFO: startup-f07e9592-0502-4a24-aff9-ef1ad4288606 started at 2023-01-28 17:39:59 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container busybox ready: false, restart count 0 Jan 28 17:44:38.672: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6vfpn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:38.672: INFO: pod-update-224dfa31-762a-41f6-8811-10c6c77b136a started at 2023-01-28 17:39:37 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container nginx ready: false, restart count 0 Jan 28 17:44:38.672: INFO: pod-7824b9b1-2ebe-4f4f-80f3-fe86d5163ca8 started at 2023-01-28 17:40:09 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container test-container ready: false, restart count 0 Jan 28 17:44:38.672: INFO: cilium-wlwtf started at 2023-01-28 17:22:12 +0000 UTC (1+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:44:38.672: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:44:38.672: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mkb65 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:44:38.672: INFO: httpd started at 2023-01-28 17:39:54 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container httpd ready: false, restart count 0 Jan 28 17:44:38.672: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-9jldw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:38.672: INFO: master started at 2023-01-28 17:44:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container cntr ready: false, restart count 0 Jan 28 17:44:38.672: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-fgg6g started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:38.672: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-ch9br started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:38.672: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:39.427: INFO: Latency metrics for node i-03342b7c6ebddc1c6 Jan 28 17:44:39.427: INFO: Logging node info for node i-0550ecdc8b2275df1 Jan 28 17:44:39.642: INFO: Node Info: &Node{ObjectMeta:{i-0550ecdc8b2275df1 e2f1fe4c-911f-4da0-9481-51f08c3e654e 33948 0 2023-01-28 17:20:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0550ecdc8b2275df1 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.60.182 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0550ecdc8b2275df1"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:20:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-01-28 17:20:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-01-28 17:21:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-28 17:22:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0550ecdc8b2275df1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:21:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.182,},NodeAddress{Type:ExternalIP,Address:52.221.229.242,},NodeAddress{Type:InternalDNS,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-229-242.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec22a890f5274dd8ae5c60b32c44aae9,SystemUUID:ec22a890-f527-4dd8-ae5c-60b32c44aae9,BootID:83ea7fa2-7c97-4062-a52f-a856b154a02c,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:231502799,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64@sha256:6db2761074bae8385aa601d4b45ab065893cfdc258d8b2593fc56698c44dbee5 k8s.gcr.io/kube-apiserver-amd64:v1.24.10 registry.k8s.io/kube-apiserver-amd64:v1.24.10],SizeBytes:131733971,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64@sha256:8867df47c332bae8dfd061b4d4230cd0febcc5a8a427ecb2e8f70b33ffc6d732 k8s.gcr.io/kube-controller-manager-amd64:v1.24.10 registry.k8s.io/kube-controller-manager-amd64:v1.24.10],SizeBytes:121342265,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64@sha256:a589f10748fa227ea6ebf0730126f3f192cbdbb2c7f96a9d4685cabb86eb1cbc k8s.gcr.io/kube-scheduler-amd64:v1.24.10 registry.k8s.io/kube-scheduler-amd64:v1.24.10],SizeBytes:52751160,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.2],SizeBytes:43830714,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.2],SizeBytes:42821730,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:26802430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:245e81c2ed53b5508a6f687f3ab0c724b9ab31e6a5fa5e6d2f814981ab6ce55f registry.k8s.io/provider-aws/cloud-controller-manager:v1.24.3],SizeBytes:18282737,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.2],SizeBytes:4965809,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:44:39.642: INFO: Logging kubelet events for node i-0550ecdc8b2275df1 Jan 28 17:44:39.861: INFO: Logging pods the kubelet thinks is on node i-0550ecdc8b2275df1 Jan 28 17:44:40.084: INFO: etcd-manager-events-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:40.084: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:44:40.084: INFO: etcd-manager-main-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:40.084: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:44:40.084: INFO: kube-apiserver-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+2 container statuses recorded) Jan 28 17:44:40.084: INFO: Container healthcheck ready: true, restart count 0 Jan 28 17:44:40.084: INFO: Container kube-apiserver ready: true, restart count 0 Jan 28 17:44:40.084: INFO: kube-controller-manager-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:40.084: INFO: Container kube-controller-manager ready: true, restart count 3 Jan 28 17:44:40.084: INFO: kops-controller-gll72 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:40.084: INFO: Container kops-controller ready: true, restart count 0 Jan 28 17:44:40.084: INFO: cilium-4trt7 started at 2023-01-28 17:21:03 +0000 UTC (1+1 container statuses recorded) Jan 28 17:44:40.084: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:44:40.084: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:44:40.084: INFO: aws-cloud-controller-manager-w6p6f started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:40.084: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Jan 28 17:44:40.084: INFO: cilium-operator-8d8478c74-2vsdk started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:40.084: INFO: Container cilium-operator ready: true, restart count 0 Jan 28 17:44:40.084: INFO: ebs-csi-controller-779575c67d-kx2cv started at 2023-01-28 17:21:03 +0000 UTC (0+5 container statuses recorded) Jan 28 17:44:40.084: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:44:40.084: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:44:40.084: INFO: Container csi-resizer ready: true, restart count 0 Jan 28 17:44:40.084: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:44:40.084: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:44:40.084: INFO: kube-scheduler-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:40.084: INFO: Container kube-scheduler ready: true, restart count 0 Jan 28 17:44:40.084: INFO: ebs-csi-node-stxgw started at 2023-01-28 17:21:03 +0000 UTC (0+3 container statuses recorded) Jan 28 17:44:40.084: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:44:40.084: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:44:40.084: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:44:40.084: INFO: dns-controller-7d4957564c-nd8s8 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:40.084: INFO: Container dns-controller ready: true, restart count 0 Jan 28 17:44:40.774: INFO: Latency metrics for node i-0550ecdc8b2275df1 Jan 28 17:44:40.774: INFO: Logging node info for node i-06d4799a5db582858 Jan 28 17:44:40.988: INFO: Node Info: &Node{ObjectMeta:{i-06d4799a5db582858 aacf1bf6-d2bd-4f61-aad7-83531f5d9ba1 33004 0 2023-01-28 17:22:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-06d4799a5db582858 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-06d4799a5db582858 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.139 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06d4799a5db582858"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {e2e.test Update v1 2023-01-28 17:39:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:40:14 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 17:40:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-06d4799a5db582858,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.139,},NodeAddress{Type:ExternalIP,Address:18.143.137.61,},NodeAddress{Type:InternalDNS,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-143-137-61.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6f00e5a83cc422a6226665122e9d,SystemUUID:ec2e6f00-e5a8-3cc4-22a6-226665122e9d,BootID:3bfeb6d1-b807-4cfa-a0a7-954aef9e3823,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc kubernetes.io/csi/ebs.csi.aws.com^vol-0dff02acc75d59bc3],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0dff02acc75d59bc3,DevicePath:,},},Config:nil,},} Jan 28 17:44:40.989: INFO: Logging kubelet events for node i-06d4799a5db582858 Jan 28 17:44:41.205: INFO: Logging pods the kubelet thinks is on node i-06d4799a5db582858 Jan 28 17:44:41.436: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-t9lsx started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.436: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:41.436: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-r9lbh started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.436: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:41.436: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-slm8r started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.436: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:41.436: INFO: emptydir-injector started at 2023-01-28 17:40:31 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.436: INFO: Container emptydir-injector ready: false, restart count 0 Jan 28 17:44:41.436: INFO: csi-mockplugin-attacher-0 started at 2023-01-28 17:40:56 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.436: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:44:41.436: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-zvdzg started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.436: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:41.436: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mkkk7 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.436: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:41.436: INFO: pod-configmaps-7cd33bfb-d4f5-467e-946d-cdf07018d8cf started at 2023-01-28 17:43:01 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.436: INFO: Container env-test ready: false, restart count 0 Jan 28 17:44:41.436: INFO: httpd started at 2023-01-28 17:44:35 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.437: INFO: Container httpd ready: false, restart count 0 Jan 28 17:44:41.437: INFO: up-down-2-vg8d2 started at 2023-01-28 17:38:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.437: INFO: Container up-down-2 ready: true, restart count 0 Jan 28 17:44:41.437: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-dgjbr started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.437: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:44:41.437: INFO: cilium-8cb5h started at 2023-01-28 17:22:14 +0000 UTC (1+1 container statuses recorded) Jan 28 17:44:41.437: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:44:41.437: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:44:41.437: INFO: ss-0 started at 2023-01-28 17:40:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.437: INFO: Container webserver ready: false, restart count 0 Jan 28 17:44:41.437: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-bdtqt started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.437: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:41.437: INFO: pod-server-1 started at 2023-01-28 17:40:22 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.437: INFO: Container agnhost-container ready: false, restart count 0 Jan 28 17:44:41.437: INFO: csi-mockplugin-0 started at 2023-01-28 17:40:55 +0000 UTC (0+3 container statuses recorded) Jan 28 17:44:41.437: INFO: Container csi-provisioner ready: false, restart count 0 Jan 28 17:44:41.437: INFO: Container driver-registrar ready: false, restart count 0 Jan 28 17:44:41.437: INFO: Container mock ready: false, restart count 0 Jan 28 17:44:41.437: INFO: ebs-csi-node-ttfmk started at 2023-01-28 17:22:14 +0000 UTC (0+3 container statuses recorded) Jan 28 17:44:41.437: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:44:41.437: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:44:41.437: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:44:41.437: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-lw565 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.437: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:41.437: INFO: busybox-702a2763-bdb2-44b8-ae2d-94e4cd0026af started at 2023-01-28 17:42:05 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.437: INFO: Container busybox ready: false, restart count 0 Jan 28 17:44:41.437: INFO: adopt-release-g5t8f started at 2023-01-28 17:42:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.437: INFO: Container c ready: false, restart count 0 Jan 28 17:44:41.437: INFO: rs-tbjch started at 2023-01-28 17:40:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.437: INFO: Container donothing ready: false, restart count 0 Jan 28 17:44:41.437: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-jq2xw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.437: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:41.437: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-7dx5z started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.437: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:41.437: INFO: pod-270e9e95-df4d-4169-9c6c-9329994f97a6 started at 2023-01-28 17:43:18 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.437: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:44:41.437: INFO: up-down-3-r87lj started at 2023-01-28 17:39:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:41.437: INFO: Container up-down-3 ready: true, restart count 0 Jan 28 17:44:42.190: INFO: Latency metrics for node i-06d4799a5db582858 Jan 28 17:44:42.190: INFO: Logging node info for node i-0d3b431203a4a372c Jan 28 17:44:42.405: INFO: Node Info: &Node{ObjectMeta:{i-0d3b431203a4a372c b69212ba-b617-41e9-91e3-ee5929b727b2 33066 0 2023-01-28 17:22:12 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d3b431203a4a372c kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0d3b431203a4a372c topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.33.52 csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-5984":"i-0d3b431203a4a372c","ebs.csi.aws.com":"i-0d3b431203a4a372c"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {e2e.test Update v1 2023-01-28 17:39:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:40:44 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 17:40:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0d3b431203a4a372c,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:45 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:45 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:45 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:40:45 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.52,},NodeAddress{Type:ExternalIP,Address:13.229.100.227,},NodeAddress{Type:InternalDNS,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-229-100-227.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2fe0eb6b322558c1427d0b9ac3c84d,SystemUUID:ec2fe0eb-6b32-2558-c142-7d0b9ac3c84d,BootID:47c658c8-8593-40bf-a02e-622027452e1a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-5984^de788a96-9f32-11ed-a845-3a68daf3203b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-5984^de788a96-9f32-11ed-a845-3a68daf3203b,DevicePath:,},},Config:nil,},} Jan 28 17:44:42.405: INFO: Logging kubelet events for node i-0d3b431203a4a372c Jan 28 17:44:42.622: INFO: Logging pods the kubelet thinks is on node i-0d3b431203a4a372c Jan 28 17:44:42.847: INFO: rs-n86z8 started at 2023-01-28 17:40:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Container donothing ready: false, restart count 0 Jan 28 17:44:42.848: INFO: csi-mockplugin-0 started at 2023-01-28 17:40:16 +0000 UTC (0+3 container statuses recorded) Jan 28 17:44:42.848: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:44:42.848: INFO: Container driver-registrar ready: true, restart count 0 Jan 28 17:44:42.848: INFO: Container mock ready: true, restart count 0 Jan 28 17:44:42.848: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-45pqn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:44:42.848: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-z6hs2 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:44:42.848: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-7bq4k started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:42.848: INFO: csi-mockplugin-attacher-0 started at 2023-01-28 17:40:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:44:42.848: INFO: pod-client started at 2023-01-28 17:39:07 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Container pod-client ready: true, restart count 0 Jan 28 17:44:42.848: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-kmzfw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:42.848: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-fzmt8 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:42.848: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6mbxw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:44:42.848: INFO: ebs-csi-node-j2pwn started at 2023-01-28 17:22:13 +0000 UTC (0+3 container statuses recorded) Jan 28 17:44:42.848: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:44:42.848: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:44:42.848: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:44:42.848: INFO: coredns-6dd78b7459-t8lf4 started at 2023-01-28 17:22:48 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Container coredns ready: true, restart count 0 Jan 28 17:44:42.848: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-g7jvs started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:42.848: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-f8tsq started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:42.848: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-hnjmr started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:42.848: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mmn5l started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:42.848: INFO: cilium-cs2hk started at 2023-01-28 17:22:13 +0000 UTC (1+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:44:42.848: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:44:42.848: INFO: pvc-volume-tester-c8gr8 started at 2023-01-28 17:40:35 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:42.848: INFO: Container volume-tester ready: true, restart count 0 Jan 28 17:44:42.848: INFO: csi-mockplugin-0 started at 2023-01-28 17:41:21 +0000 UTC (0+4 container statuses recorded) Jan 28 17:44:42.848: INFO: Container busybox ready: false, restart count 0 Jan 28 17:44:42.848: INFO: Container csi-provisioner ready: false, restart count 0 Jan 28 17:44:42.848: INFO: Container driver-registrar ready: false, restart count 0 Jan 28 17:44:42.848: INFO: Container mock ready: false, restart count 0 Jan 28 17:44:43.590: INFO: Latency metrics for node i-0d3b431203a4a372c Jan 28 17:44:43.590: INFO: Logging node info for node i-0e10813507581b097 Jan 28 17:44:43.805: INFO: Node Info: &Node{ObjectMeta:{i-0e10813507581b097 8af9c758-a6f6-4a0d-b634-4d99ed71b929 32762 0 2023-01-28 17:22:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e10813507581b097 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0e10813507581b097 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.58 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e10813507581b097"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {e2e.test Update v1 2023-01-28 17:39:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:40:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0e10813507581b097,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:40:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.58,},NodeAddress{Type:ExternalIP,Address:54.254.187.204,},NodeAddress{Type:InternalDNS,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-187-204.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2922b3ab12fd264b764b33da3e62c2,SystemUUID:ec2922b3-ab12-fd26-4b76-4b33da3e62c2,BootID:07cb73db-252f-45aa-8a98-af3a42c0313b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:44:43.805: INFO: Logging kubelet events for node i-0e10813507581b097 Jan 28 17:44:44.021: INFO: Logging pods the kubelet thinks is on node i-0e10813507581b097 Jan 28 17:44:44.245: INFO: coredns-autoscaler-5f76bf6c7b-nnmkq started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container autoscaler ready: true, restart count 0 Jan 28 17:44:44.245: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-89jgx started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:44.245: INFO: pod-server-2 started at 2023-01-28 17:39:35 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container agnhost-container ready: false, restart count 0 Jan 28 17:44:44.245: INFO: up-down-3-bmxqn started at 2023-01-28 17:39:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container up-down-3 ready: false, restart count 0 Jan 28 17:44:44.245: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-rl6v2 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:44.245: INFO: cilium-mbfkc started at 2023-01-28 17:22:08 +0000 UTC (1+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:44:44.245: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:44:44.245: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-62lqj started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:44:44.245: INFO: pod-client started at 2023-01-28 17:40:19 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container pod-client ready: true, restart count 0 Jan 28 17:44:44.245: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mscfw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:44:44.245: INFO: up-down-2-2j47t started at 2023-01-28 17:38:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container up-down-2 ready: true, restart count 0 Jan 28 17:44:44.245: INFO: ebs-csi-node-k6rdn started at 2023-01-28 17:22:08 +0000 UTC (0+3 container statuses recorded) Jan 28 17:44:44.245: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:44:44.245: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:44:44.245: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:44:44.245: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-vnxp9 started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:44.245: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-2j7n9 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:44:44.245: INFO: pod-736ddca1-ca58-4462-8ac4-c2357b897d98 started at 2023-01-28 17:40:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:44:44.245: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-4hpf7 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:44.245: INFO: hostexec-i-0e10813507581b097-zj6lj started at 2023-01-28 17:40:20 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:44:44.245: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-x5gs8 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:44.245: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-22rht started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:44:44.245: INFO: coredns-6dd78b7459-qk7rr started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container coredns ready: true, restart count 0 Jan 28 17:44:44.245: INFO: test-ss-0 started at 2023-01-28 17:39:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container webserver ready: true, restart count 0 Jan 28 17:44:44.245: INFO: pod-server-1 started at 2023-01-28 17:39:12 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:44:44.245: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-9hkr5 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:44:44.245: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:44:44.961: INFO: Latency metrics for node i-0e10813507581b097 Jan 28 17:44:44.961: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "conntrack-3826" for this suite.
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-network\]\sServices\sshould\sbe\sable\sto\sup\sand\sdown\sservices$'
test/e2e/network/service.go:1045 Jan 28 17:45:06.588: failed to create replication controller with service: up-down-3 in the namespace: services-3650 Unexpected error: <*errors.errorString | 0xc001d5c650>: { s: "only 2 pods started out of 3", } only 2 pods started out of 3 occurred test/e2e/network/service.go:1078from junit_22.xml
[BeforeEach] [sig-network] Services test/e2e/framework/framework.go:187 �[1mSTEP�[0m: Creating a kubernetes client Jan 28 17:38:41.821: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename services �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-network] Services test/e2e/network/service.go:758 [It] should be able to up and down services test/e2e/network/service.go:1045 �[1mSTEP�[0m: creating up-down-1 in namespace services-3650 �[1mSTEP�[0m: creating service up-down-1 in namespace services-3650 �[1mSTEP�[0m: creating replication controller up-down-1 in namespace services-3650 I0128 17:38:43.762396 6772 runners.go:193] Created replication controller with name: up-down-1, namespace: services-3650, replica count: 3 I0128 17:38:47.013633 6772 runners.go:193] up-down-1 Pods: 3 out of 3 created, 3 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady �[1mSTEP�[0m: creating up-down-2 in namespace services-3650 �[1mSTEP�[0m: creating service up-down-2 in namespace services-3650 �[1mSTEP�[0m: creating replication controller up-down-2 in namespace services-3650 I0128 17:38:47.666781 6772 runners.go:193] Created replication controller with name: up-down-2, namespace: services-3650, replica count: 3 I0128 17:38:50.918165 6772 runners.go:193] up-down-2 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:38:53.918466 6772 runners.go:193] up-down-2 Pods: 3 out of 3 created, 3 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady �[1mSTEP�[0m: verifying service up-down-1 is up Jan 28 17:38:54.132: INFO: Creating new host exec pod Jan 28 17:38:54.564: INFO: The status of Pod verify-service-up-host-exec-pod is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:38:56.778: INFO: The status of Pod verify-service-up-host-exec-pod is Running (Ready = true) Jan 28 17:38:56.779: INFO: Creating new exec pod �[1mSTEP�[0m: verifying service has 3 reachable backends Jan 28 17:39:01.426: INFO: Executing cmd "for i in $(seq 1 150); do wget -q -O - -T 1 http://100.71.167.231:80 2>&1 || true; echo; done" in pod services-3650/verify-service-up-host-exec-pod Jan 28 17:39:01.426: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-3650 exec verify-service-up-host-exec-pod -- /bin/sh -x -c for i in $(seq 1 150); do wget -q -O - -T 1 http://100.71.167.231:80 2>&1 || true; echo; done' Jan 28 17:39:03.908: INFO: stderr: "+ seq 1 150\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n" Jan 28 17:39:03.908: INFO: stdout: "up-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-z2x8k\n" Jan 28 17:39:03.908: INFO: Executing cmd "for i in $(seq 1 150); do wget -q -O - -T 1 http://100.71.167.231:80 2>&1 || true; echo; done" in pod services-3650/verify-service-up-exec-pod-c55vw Jan 28 17:39:03.908: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-3650 exec verify-service-up-exec-pod-c55vw -- /bin/sh -x -c for i in $(seq 1 150); do wget -q -O - -T 1 http://100.71.167.231:80 2>&1 || true; echo; done' Jan 28 17:39:06.422: INFO: stderr: "+ seq 1 150\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n+ wget -q -O - -T 1 http://100.71.167.231:80\n+ echo\n" Jan 28 17:39:06.423: INFO: stdout: "up-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-z2x8k\nup-down-1-6mf57\nup-down-1-6mf57\nup-down-1-mkl7s\nup-down-1-mkl7s\nup-down-1-6mf57\nup-down-1-z2x8k\nup-down-1-z2x8k\nup-down-1-mkl7s\nup-down-1-6mf57\n" �[1mSTEP�[0m: Deleting pod verify-service-up-host-exec-pod in namespace services-3650 �[1mSTEP�[0m: Deleting pod verify-service-up-exec-pod-c55vw in namespace services-3650 �[1mSTEP�[0m: verifying service up-down-2 is up Jan 28 17:39:06.877: INFO: Creating new host exec pod Jan 28 17:39:07.310: INFO: The status of Pod verify-service-up-host-exec-pod is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:09.525: INFO: The status of Pod verify-service-up-host-exec-pod is Running (Ready = true) Jan 28 17:39:09.525: INFO: Creating new exec pod �[1mSTEP�[0m: verifying service has 3 reachable backends Jan 28 17:39:14.176: INFO: Executing cmd "for i in $(seq 1 150); do wget -q -O - -T 1 http://100.68.98.198:80 2>&1 || true; echo; done" in pod services-3650/verify-service-up-host-exec-pod Jan 28 17:39:14.176: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-3650 exec verify-service-up-host-exec-pod -- /bin/sh -x -c for i in $(seq 1 150); do wget -q -O - -T 1 http://100.68.98.198:80 2>&1 || true; echo; done' Jan 28 17:39:19.378: INFO: stderr: "+ seq 1 150\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n" Jan 28 17:39:19.378: INFO: stdout: "up-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\n" Jan 28 17:39:19.378: INFO: Executing cmd "for i in $(seq 1 150); do wget -q -O - -T 1 http://100.68.98.198:80 2>&1 || true; echo; done" in pod services-3650/verify-service-up-exec-pod-5vgvd Jan 28 17:39:19.378: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-3650 exec verify-service-up-exec-pod-5vgvd -- /bin/sh -x -c for i in $(seq 1 150); do wget -q -O - -T 1 http://100.68.98.198:80 2>&1 || true; echo; done' Jan 28 17:39:22.992: INFO: stderr: "+ seq 1 150\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n" Jan 28 17:39:22.992: INFO: stdout: "up-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\n" �[1mSTEP�[0m: Deleting pod verify-service-up-host-exec-pod in namespace services-3650 �[1mSTEP�[0m: Deleting pod verify-service-up-exec-pod-5vgvd in namespace services-3650 �[1mSTEP�[0m: stopping service up-down-1 �[1mSTEP�[0m: deleting ReplicationController up-down-1 in namespace services-3650, will wait for the garbage collector to delete the pods Jan 28 17:39:24.113: INFO: Deleting ReplicationController up-down-1 took: 215.855401ms Jan 28 17:39:24.214: INFO: Terminating ReplicationController up-down-1 pods took: 101.146955ms �[1mSTEP�[0m: verifying service up-down-1 is not up Jan 28 17:39:29.941: INFO: Creating new host exec pod Jan 28 17:39:30.374: INFO: The status of Pod verify-service-down-host-exec-pod is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:32.592: INFO: The status of Pod verify-service-down-host-exec-pod is Running (Ready = true) Jan 28 17:39:32.592: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-3650 exec verify-service-down-host-exec-pod -- /bin/sh -x -c curl -g -s --connect-timeout 2 http://100.71.167.231:80 && echo service-down-failed' Jan 28 17:39:36.707: INFO: rc: 28 Jan 28 17:39:36.707: INFO: error while kubectl execing "curl -g -s --connect-timeout 2 http://100.71.167.231:80 && echo service-down-failed" in pod services-3650/verify-service-down-host-exec-pod: error running /home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-3650 exec verify-service-down-host-exec-pod -- /bin/sh -x -c curl -g -s --connect-timeout 2 http://100.71.167.231:80 && echo service-down-failed: Command stdout: stderr: + curl -g -s --connect-timeout 2 http://100.71.167.231:80 command terminated with exit code 28 error: exit status 28 Output: �[1mSTEP�[0m: Deleting pod verify-service-down-host-exec-pod in namespace services-3650 �[1mSTEP�[0m: verifying service up-down-2 is still up Jan 28 17:39:36.928: INFO: Creating new host exec pod Jan 28 17:39:37.358: INFO: The status of Pod verify-service-up-host-exec-pod is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:39.573: INFO: The status of Pod verify-service-up-host-exec-pod is Running (Ready = true) Jan 28 17:39:39.574: INFO: Creating new exec pod �[1mSTEP�[0m: verifying service has 3 reachable backends Jan 28 17:39:42.219: INFO: Executing cmd "for i in $(seq 1 150); do wget -q -O - -T 1 http://100.68.98.198:80 2>&1 || true; echo; done" in pod services-3650/verify-service-up-host-exec-pod Jan 28 17:39:42.219: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-3650 exec verify-service-up-host-exec-pod -- /bin/sh -x -c for i in $(seq 1 150); do wget -q -O - -T 1 http://100.68.98.198:80 2>&1 || true; echo; done' Jan 28 17:39:44.762: INFO: stderr: "+ seq 1 150\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n" Jan 28 17:39:44.762: INFO: stdout: "up-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\n" Jan 28 17:39:44.762: INFO: Executing cmd "for i in $(seq 1 150); do wget -q -O - -T 1 http://100.68.98.198:80 2>&1 || true; echo; done" in pod services-3650/verify-service-up-exec-pod-8xccx Jan 28 17:39:44.762: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/5841c34c-9f2f-11ed-9a0c-f25a29267a91/kubectl --server=https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-3650 exec verify-service-up-exec-pod-8xccx -- /bin/sh -x -c for i in $(seq 1 150); do wget -q -O - -T 1 http://100.68.98.198:80 2>&1 || true; echo; done' Jan 28 17:39:47.127: INFO: stderr: "+ seq 1 150\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n+ wget -q -O - -T 1 http://100.68.98.198:80\n+ echo\n" Jan 28 17:39:47.127: INFO: stdout: "up-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-rw86m\nup-down-2-vg8d2\nup-down-2-2j47t\nup-down-2-rw86m\nup-down-2-vg8d2\n" �[1mSTEP�[0m: Deleting pod verify-service-up-host-exec-pod in namespace services-3650 �[1mSTEP�[0m: Deleting pod verify-service-up-exec-pod-8xccx in namespace services-3650 �[1mSTEP�[0m: creating service up-down-3 in namespace services-3650 �[1mSTEP�[0m: creating service up-down-3 in namespace services-3650 �[1mSTEP�[0m: creating replication controller up-down-3 in namespace services-3650 I0128 17:39:48.023070 6772 runners.go:193] Created replication controller with name: up-down-3, namespace: services-3650, replica count: 3 I0128 17:39:51.274316 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:39:54.274591 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:39:57.275017 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:00.276286 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:03.276589 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 1 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:06.277023 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:09.277334 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:12.278343 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:15.278712 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:18.281193 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:21.281568 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:24.282823 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:27.284121 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:30.285364 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:33.285972 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:36.286319 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:39.286633 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:42.287005 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:45.287367 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:48.289019 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:51.289447 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:54.289747 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:40:57.290179 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:00.290699 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:03.291654 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:06.291985 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:09.292291 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:12.292668 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:15.293747 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:18.296068 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:21.296449 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:24.297292 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:27.299597 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:30.299946 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:33.300701 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:36.301052 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:39.301377 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:42.302369 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:45.304285 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:48.306650 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:51.307650 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:54.307982 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:41:57.308200 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:00.309630 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:03.310727 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:06.311916 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:09.312186 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:12.314596 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:15.314962 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:18.317355 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:21.319873 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:24.320968 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:27.321327 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:30.321670 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:33.322820 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:36.325316 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:39.325630 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:42.327915 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:45.328254 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:48.328594 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:51.329141 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:54.329548 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:42:57.331874 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:00.333321 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:03.333606 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:06.333981 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:09.335153 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:12.337184 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:15.338800 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:18.339103 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:21.341006 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:24.341627 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:27.344119 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:30.345651 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:33.346773 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:36.347225 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:39.347719 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:42.348122 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:45.349807 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:48.350756 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:51.353480 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:54.354098 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:43:57.354589 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:00.355662 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:03.355949 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:06.356244 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:09.356556 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:12.358867 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:15.360550 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:18.361463 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:21.361871 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:24.362235 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:27.362587 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:30.362927 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:33.363984 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:36.364588 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:39.364900 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:42.365137 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:45.365477 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:48.365819 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:51.366164 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:54.366492 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:44:57.367604 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:45:00.368021 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:45:03.368613 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:45:06.371322 6772 runners.go:193] up-down-3 Pods: 3 out of 3 created, 2 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0128 17:45:06.587620 6772 runners.go:193] Pod up-down-3-bmxqn i-0e10813507581b097 Pending <nil> I0128 17:45:06.587771 6772 runners.go:193] Pod up-down-3-r87lj i-06d4799a5db582858 Running <nil> I0128 17:45:06.587787 6772 runners.go:193] Pod up-down-3-rcvxq i-03342b7c6ebddc1c6 Running <nil> Jan 28 17:45:06.588: FAIL: failed to create replication controller with service: up-down-3 in the namespace: services-3650 Unexpected error: <*errors.errorString | 0xc001d5c650>: { s: "only 2 pods started out of 3", } only 2 pods started out of 3 occurred Full Stack Trace k8s.io/kubernetes/test/e2e/network.glob..func25.8() test/e2e/network/service.go:1078 +0x7ac k8s.io/kubernetes/test/e2e.RunE2ETests(0x25761d7?) test/e2e/e2e.go:130 +0x6bb k8s.io/kubernetes/test/e2e.TestE2E(0x24e52d9?) test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc000b9a000, 0x741f9a8) /usr/local/go/src/testing/testing.go:1446 +0x10b created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1493 +0x35f [AfterEach] [sig-network] Services test/e2e/framework/framework.go:188 �[1mSTEP�[0m: Collecting events from namespace "services-3650". �[1mSTEP�[0m: Found 91 events. Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:43 +0000 UTC - event for up-down-1: {replication-controller } SuccessfulCreate: Created pod: up-down-1-z2x8k Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:43 +0000 UTC - event for up-down-1: {replication-controller } SuccessfulCreate: Created pod: up-down-1-6mf57 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:43 +0000 UTC - event for up-down-1: {replication-controller } SuccessfulCreate: Created pod: up-down-1-mkl7s Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:43 +0000 UTC - event for up-down-1-6mf57: {default-scheduler } Scheduled: Successfully assigned services-3650/up-down-1-6mf57 to i-0e10813507581b097 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:43 +0000 UTC - event for up-down-1-mkl7s: {default-scheduler } Scheduled: Successfully assigned services-3650/up-down-1-mkl7s to i-06d4799a5db582858 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:43 +0000 UTC - event for up-down-1-z2x8k: {default-scheduler } Scheduled: Successfully assigned services-3650/up-down-1-z2x8k to i-03342b7c6ebddc1c6 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:45 +0000 UTC - event for up-down-1-6mf57: {kubelet i-0e10813507581b097} Started: Started container up-down-1 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:45 +0000 UTC - event for up-down-1-6mf57: {kubelet i-0e10813507581b097} Created: Created container up-down-1 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:45 +0000 UTC - event for up-down-1-6mf57: {kubelet i-0e10813507581b097} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:45 +0000 UTC - event for up-down-1-mkl7s: {kubelet i-06d4799a5db582858} Created: Created container up-down-1 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:45 +0000 UTC - event for up-down-1-mkl7s: {kubelet i-06d4799a5db582858} Started: Started container up-down-1 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:45 +0000 UTC - event for up-down-1-mkl7s: {kubelet i-06d4799a5db582858} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:45 +0000 UTC - event for up-down-1-z2x8k: {kubelet i-03342b7c6ebddc1c6} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:45 +0000 UTC - event for up-down-1-z2x8k: {kubelet i-03342b7c6ebddc1c6} Created: Created container up-down-1 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:45 +0000 UTC - event for up-down-1-z2x8k: {kubelet i-03342b7c6ebddc1c6} Started: Started container up-down-1 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:47 +0000 UTC - event for up-down-2: {replication-controller } SuccessfulCreate: Created pod: up-down-2-vg8d2 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:47 +0000 UTC - event for up-down-2: {replication-controller } SuccessfulCreate: Created pod: up-down-2-rw86m Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:47 +0000 UTC - event for up-down-2: {replication-controller } SuccessfulCreate: Created pod: up-down-2-2j47t Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:47 +0000 UTC - event for up-down-2-2j47t: {default-scheduler } Scheduled: Successfully assigned services-3650/up-down-2-2j47t to i-0e10813507581b097 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:47 +0000 UTC - event for up-down-2-rw86m: {default-scheduler } Scheduled: Successfully assigned services-3650/up-down-2-rw86m to i-03342b7c6ebddc1c6 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:47 +0000 UTC - event for up-down-2-vg8d2: {default-scheduler } Scheduled: Successfully assigned services-3650/up-down-2-vg8d2 to i-06d4799a5db582858 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:49 +0000 UTC - event for up-down-2-rw86m: {kubelet i-03342b7c6ebddc1c6} Created: Created container up-down-2 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:49 +0000 UTC - event for up-down-2-rw86m: {kubelet i-03342b7c6ebddc1c6} Started: Started container up-down-2 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:49 +0000 UTC - event for up-down-2-rw86m: {kubelet i-03342b7c6ebddc1c6} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:49 +0000 UTC - event for up-down-2-vg8d2: {kubelet i-06d4799a5db582858} Created: Created container up-down-2 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:49 +0000 UTC - event for up-down-2-vg8d2: {kubelet i-06d4799a5db582858} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:49 +0000 UTC - event for up-down-2-vg8d2: {kubelet i-06d4799a5db582858} Started: Started container up-down-2 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:50 +0000 UTC - event for up-down-2-2j47t: {kubelet i-0e10813507581b097} Started: Started container up-down-2 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:50 +0000 UTC - event for up-down-2-2j47t: {kubelet i-0e10813507581b097} Created: Created container up-down-2 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:50 +0000 UTC - event for up-down-2-2j47t: {kubelet i-0e10813507581b097} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:54 +0000 UTC - event for verify-service-up-host-exec-pod: {default-scheduler } Scheduled: Successfully assigned services-3650/verify-service-up-host-exec-pod to i-0e10813507581b097 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:55 +0000 UTC - event for verify-service-up-host-exec-pod: {kubelet i-0e10813507581b097} Started: Started container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:55 +0000 UTC - event for verify-service-up-host-exec-pod: {kubelet i-0e10813507581b097} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:55 +0000 UTC - event for verify-service-up-host-exec-pod: {kubelet i-0e10813507581b097} Created: Created container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:56 +0000 UTC - event for verify-service-up-exec-pod-c55vw: {default-scheduler } Scheduled: Successfully assigned services-3650/verify-service-up-exec-pod-c55vw to i-06d4799a5db582858 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:59 +0000 UTC - event for verify-service-up-exec-pod-c55vw: {kubelet i-06d4799a5db582858} Created: Created container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:38:59 +0000 UTC - event for verify-service-up-exec-pod-c55vw: {kubelet i-06d4799a5db582858} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:00 +0000 UTC - event for verify-service-up-exec-pod-c55vw: {kubelet i-06d4799a5db582858} Started: Started container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:06 +0000 UTC - event for verify-service-up-exec-pod-c55vw: {kubelet i-06d4799a5db582858} Killing: Stopping container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:06 +0000 UTC - event for verify-service-up-host-exec-pod: {kubelet i-0e10813507581b097} Killing: Stopping container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:06 +0000 UTC - event for verify-service-up-host-exec-pod: {default-scheduler } Scheduled: Successfully assigned services-3650/verify-service-up-host-exec-pod to i-0e10813507581b097 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:07 +0000 UTC - event for verify-service-up-host-exec-pod: {kubelet i-0e10813507581b097} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:07 +0000 UTC - event for verify-service-up-host-exec-pod: {kubelet i-0e10813507581b097} Created: Created container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:07 +0000 UTC - event for verify-service-up-host-exec-pod: {kubelet i-0e10813507581b097} Started: Started container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:09 +0000 UTC - event for verify-service-up-exec-pod-5vgvd: {default-scheduler } Scheduled: Successfully assigned services-3650/verify-service-up-exec-pod-5vgvd to i-06d4799a5db582858 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:12 +0000 UTC - event for verify-service-up-exec-pod-5vgvd: {kubelet i-06d4799a5db582858} Created: Created container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:12 +0000 UTC - event for verify-service-up-exec-pod-5vgvd: {kubelet i-06d4799a5db582858} Started: Started container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:12 +0000 UTC - event for verify-service-up-exec-pod-5vgvd: {kubelet i-06d4799a5db582858} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:23 +0000 UTC - event for verify-service-up-exec-pod-5vgvd: {kubelet i-06d4799a5db582858} Killing: Stopping container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:23 +0000 UTC - event for verify-service-up-host-exec-pod: {kubelet i-0e10813507581b097} Killing: Stopping container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:24 +0000 UTC - event for up-down-1-6mf57: {kubelet i-0e10813507581b097} Killing: Stopping container up-down-1 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:24 +0000 UTC - event for up-down-1-mkl7s: {kubelet i-06d4799a5db582858} Killing: Stopping container up-down-1 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:24 +0000 UTC - event for up-down-1-z2x8k: {kubelet i-03342b7c6ebddc1c6} Killing: Stopping container up-down-1 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:30 +0000 UTC - event for verify-service-down-host-exec-pod: {default-scheduler } Scheduled: Successfully assigned services-3650/verify-service-down-host-exec-pod to i-03342b7c6ebddc1c6 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:30 +0000 UTC - event for verify-service-down-host-exec-pod: {kubelet i-03342b7c6ebddc1c6} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:30 +0000 UTC - event for verify-service-down-host-exec-pod: {kubelet i-03342b7c6ebddc1c6} Created: Created container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:31 +0000 UTC - event for verify-service-down-host-exec-pod: {kubelet i-03342b7c6ebddc1c6} Started: Started container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:36 +0000 UTC - event for verify-service-down-host-exec-pod: {kubelet i-03342b7c6ebddc1c6} Killing: Stopping container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:37 +0000 UTC - event for verify-service-up-host-exec-pod: {kubelet i-06d4799a5db582858} Started: Started container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:37 +0000 UTC - event for verify-service-up-host-exec-pod: {kubelet i-06d4799a5db582858} Created: Created container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:37 +0000 UTC - event for verify-service-up-host-exec-pod: {kubelet i-06d4799a5db582858} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:37 +0000 UTC - event for verify-service-up-host-exec-pod: {default-scheduler } Scheduled: Successfully assigned services-3650/verify-service-up-host-exec-pod to i-06d4799a5db582858 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:39 +0000 UTC - event for verify-service-up-exec-pod-8xccx: {default-scheduler } Scheduled: Successfully assigned services-3650/verify-service-up-exec-pod-8xccx to i-06d4799a5db582858 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:41 +0000 UTC - event for verify-service-up-exec-pod-8xccx: {kubelet i-06d4799a5db582858} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:41 +0000 UTC - event for verify-service-up-exec-pod-8xccx: {kubelet i-06d4799a5db582858} Created: Created container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:41 +0000 UTC - event for verify-service-up-exec-pod-8xccx: {kubelet i-06d4799a5db582858} Started: Started container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:47 +0000 UTC - event for up-down-3: {replication-controller } SuccessfulCreate: Created pod: up-down-3-rcvxq Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:47 +0000 UTC - event for up-down-3: {replication-controller } SuccessfulCreate: Created pod: up-down-3-r87lj Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:47 +0000 UTC - event for up-down-3: {replication-controller } SuccessfulCreate: Created pod: up-down-3-bmxqn Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:47 +0000 UTC - event for up-down-3-bmxqn: {default-scheduler } Scheduled: Successfully assigned services-3650/up-down-3-bmxqn to i-0e10813507581b097 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:47 +0000 UTC - event for up-down-3-r87lj: {default-scheduler } Scheduled: Successfully assigned services-3650/up-down-3-r87lj to i-06d4799a5db582858 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:47 +0000 UTC - event for up-down-3-rcvxq: {default-scheduler } Scheduled: Successfully assigned services-3650/up-down-3-rcvxq to i-03342b7c6ebddc1c6 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:47 +0000 UTC - event for verify-service-up-exec-pod-8xccx: {kubelet i-06d4799a5db582858} Killing: Stopping container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:47 +0000 UTC - event for verify-service-up-host-exec-pod: {kubelet i-06d4799a5db582858} Killing: Stopping container agnhost-container Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:48 +0000 UTC - event for up-down-3-bmxqn: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8000e72ce7856eae6b01c294320b4f5e2f8305b2244a78f5e48d401e613bd50a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:48 +0000 UTC - event for up-down-3-rcvxq: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d35305fd9392ef8b9bcf42429c897fd6feb1e2bd20b8b1167d37d306882970a7": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:49 +0000 UTC - event for up-down-3-r87lj: {kubelet i-06d4799a5db582858} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:49 +0000 UTC - event for up-down-3-r87lj: {kubelet i-06d4799a5db582858} Started: Started container up-down-3 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:39:49 +0000 UTC - event for up-down-3-r87lj: {kubelet i-06d4799a5db582858} Created: Created container up-down-3 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:40:00 +0000 UTC - event for up-down-3-bmxqn: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "36dc3a86d562ff8099801b0677bf20c91791b78f01f4fd6bc002e033de24f4e5": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:06.807: INFO: At 2023-01-28 17:40:03 +0000 UTC - event for up-down-3-rcvxq: {kubelet i-03342b7c6ebddc1c6} Started: Started container up-down-3 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:40:03 +0000 UTC - event for up-down-3-rcvxq: {kubelet i-03342b7c6ebddc1c6} Created: Created container up-down-3 Jan 28 17:45:06.807: INFO: At 2023-01-28 17:40:03 +0000 UTC - event for up-down-3-rcvxq: {kubelet i-03342b7c6ebddc1c6} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:45:06.807: INFO: At 2023-01-28 17:40:13 +0000 UTC - event for up-down-3-bmxqn: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7c0ef4d267e9809065b847f66818d93d42186d6d57d1da2faef5031b98c08769": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:06.807: INFO: At 2023-01-28 17:40:26 +0000 UTC - event for up-down-3-bmxqn: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "26653d772e36c473b312cd5f0a301e8c5eda4015d45eaef6918b1894911e38af": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:06.807: INFO: At 2023-01-28 17:40:37 +0000 UTC - event for up-down-3-bmxqn: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f97ba2eaf9af2c3781da8a33bd436248f54af1a66795e7f96ffc6c9b69f664da": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:06.807: INFO: At 2023-01-28 17:40:51 +0000 UTC - event for up-down-3-bmxqn: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "73ef8b14a5a1e5a3aa6484c4ec64c31e27d054be7e3809bdd7fe4717c5e1641e": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:06.807: INFO: At 2023-01-28 17:41:03 +0000 UTC - event for up-down-3-bmxqn: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ffecd6bee8b418763f2a5a3ccf8ef0fa24f3e1ab93080b5330f98d9455dc6166": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:06.807: INFO: At 2023-01-28 17:41:15 +0000 UTC - event for up-down-3-bmxqn: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cb18a57a1d7b2d0bc80394fb8b8580ef348c2b29d0e566778f1c060964c50511": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:06.807: INFO: At 2023-01-28 17:41:27 +0000 UTC - event for up-down-3-bmxqn: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e3a8d1ed6c958ee747a4e034445948db0d02992258c937786e3e54ad4753484c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:06.807: INFO: At 2023-01-28 17:41:42 +0000 UTC - event for up-down-3-bmxqn: {kubelet i-0e10813507581b097} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6d66fa672688db578561891470b690160319cd0c1c4d4dd623263f3d5d136a8b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:45:07.024: INFO: POD NODE PHASE GRACE CONDITIONS Jan 28 17:45:07.024: INFO: up-down-2-2j47t i-0e10813507581b097 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:38:47 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:38:50 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:38:50 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:38:47 +0000 UTC }] Jan 28 17:45:07.024: INFO: up-down-2-rw86m i-03342b7c6ebddc1c6 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:38:47 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:38:49 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:38:49 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:38:47 +0000 UTC }] Jan 28 17:45:07.024: INFO: up-down-2-vg8d2 i-06d4799a5db582858 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:38:47 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:38:49 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:38:49 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:38:47 +0000 UTC }] Jan 28 17:45:07.024: INFO: up-down-3-bmxqn i-0e10813507581b097 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:47 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:47 +0000 UTC ContainersNotReady containers with unready status: [up-down-3]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:47 +0000 UTC ContainersNotReady containers with unready status: [up-down-3]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:47 +0000 UTC }] Jan 28 17:45:07.024: INFO: up-down-3-r87lj i-06d4799a5db582858 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:47 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:50 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:50 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:47 +0000 UTC }] Jan 28 17:45:07.025: INFO: up-down-3-rcvxq i-03342b7c6ebddc1c6 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:47 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:40:04 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:40:04 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:39:47 +0000 UTC }] Jan 28 17:45:07.025: INFO: Jan 28 17:45:07.900: INFO: Unable to fetch services-3650/up-down-3-bmxqn/up-down-3 logs: the server rejected our request for an unknown reason (get pods up-down-3-bmxqn) Jan 28 17:45:08.336: INFO: Logging node info for node i-03342b7c6ebddc1c6 Jan 28 17:45:08.552: INFO: Node Info: &Node{ObjectMeta:{i-03342b7c6ebddc1c6 2b79525a-8646-4141-9334-8d4a7a592265 32812 0 2023-01-28 17:22:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03342b7c6ebddc1c6 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-03342b7c6ebddc1c6 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.39.74 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03342b7c6ebddc1c6"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:37:15 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 17:37:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {e2e.test Update v1 2023-01-28 17:39:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-03342b7c6ebddc1c6,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:13 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:13 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:13 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:40:13 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.74,},NodeAddress{Type:ExternalIP,Address:52.221.218.125,},NodeAddress{Type:InternalDNS,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-218-125.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bb29c4b9b2fc4087668bfcd056585,SystemUUID:ec2bb29c-4b9b-2fc4-0876-68bfcd056585,BootID:042ee65a-a182-44df-9f74-80b116af3725,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-02c58416f403003e0],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02c58416f403003e0,DevicePath:,},},Config:nil,},} Jan 28 17:45:08.553: INFO: Logging kubelet events for node i-03342b7c6ebddc1c6 Jan 28 17:45:08.771: INFO: Logging pods the kubelet thinks is on node i-03342b7c6ebddc1c6 Jan 28 17:45:09.002: INFO: pod-d024b2fd-b2af-4707-9028-d6c200999384 started at 2023-01-28 17:44:57 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:45:09.002: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-fgg6g started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:09.002: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-ch9br started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:09.002: INFO: master started at 2023-01-28 17:44:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container cntr ready: false, restart count 0 Jan 28 17:45:09.002: INFO: pod-23984187-216b-4e72-b531-729449648f82 started at 2023-01-28 17:43:00 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container test-container ready: false, restart count 0 Jan 28 17:45:09.002: INFO: update-demo-nautilus-vl9xg started at 2023-01-28 17:44:51 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container update-demo ready: false, restart count 0 Jan 28 17:45:09.002: INFO: ss-1 started at 2023-01-28 17:39:40 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container webserver ready: false, restart count 0 Jan 28 17:45:09.002: INFO: test-ss-1 started at 2023-01-28 17:40:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container webserver ready: false, restart count 0 Jan 28 17:45:09.002: INFO: adopt-release-56c4c started at 2023-01-28 17:42:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container c ready: false, restart count 0 Jan 28 17:45:09.002: INFO: rs-4ztpt started at 2023-01-28 17:40:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container donothing ready: true, restart count 0 Jan 28 17:45:09.002: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6b7pn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:09.002: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mnncj started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:09.002: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-kcssq started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:09.002: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-57xvw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:09.002: INFO: up-down-2-rw86m started at 2023-01-28 17:38:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container up-down-2 ready: true, restart count 0 Jan 28 17:45:09.002: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-jqzfn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:09.002: INFO: pod-secrets-49e03bcd-e201-4e96-8caa-cef1cb071d5d started at 2023-01-28 17:40:18 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container secret-volume-test ready: false, restart count 0 Jan 28 17:45:09.002: INFO: ebs-csi-node-xxvv5 started at 2023-01-28 17:22:12 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:09.002: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:09.002: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:09.002: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:45:09.002: INFO: up-down-3-rcvxq started at 2023-01-28 17:39:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container up-down-3 ready: true, restart count 0 Jan 28 17:45:09.002: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6vfpn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:09.002: INFO: pod-7824b9b1-2ebe-4f4f-80f3-fe86d5163ca8 started at 2023-01-28 17:40:09 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container test-container ready: false, restart count 0 Jan 28 17:45:09.002: INFO: cilium-wlwtf started at 2023-01-28 17:22:12 +0000 UTC (1+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:45:09.002: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:45:09.002: INFO: hostexec-i-03342b7c6ebddc1c6-s4cmv started at 2023-01-28 17:44:49 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:45:09.002: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-9jldw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:09.002: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mkb65 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:09.002: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:09.748: INFO: Latency metrics for node i-03342b7c6ebddc1c6 Jan 28 17:45:09.748: INFO: Logging node info for node i-0550ecdc8b2275df1 Jan 28 17:45:09.964: INFO: Node Info: &Node{ObjectMeta:{i-0550ecdc8b2275df1 e2f1fe4c-911f-4da0-9481-51f08c3e654e 33948 0 2023-01-28 17:20:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0550ecdc8b2275df1 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.60.182 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0550ecdc8b2275df1"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:20:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-01-28 17:20:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-01-28 17:21:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-28 17:22:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0550ecdc8b2275df1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:43:16 +0000 UTC,LastTransitionTime:2023-01-28 17:21:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.182,},NodeAddress{Type:ExternalIP,Address:52.221.229.242,},NodeAddress{Type:InternalDNS,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-229-242.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec22a890f5274dd8ae5c60b32c44aae9,SystemUUID:ec22a890-f527-4dd8-ae5c-60b32c44aae9,BootID:83ea7fa2-7c97-4062-a52f-a856b154a02c,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:231502799,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64@sha256:6db2761074bae8385aa601d4b45ab065893cfdc258d8b2593fc56698c44dbee5 k8s.gcr.io/kube-apiserver-amd64:v1.24.10 registry.k8s.io/kube-apiserver-amd64:v1.24.10],SizeBytes:131733971,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64@sha256:8867df47c332bae8dfd061b4d4230cd0febcc5a8a427ecb2e8f70b33ffc6d732 k8s.gcr.io/kube-controller-manager-amd64:v1.24.10 registry.k8s.io/kube-controller-manager-amd64:v1.24.10],SizeBytes:121342265,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64@sha256:a589f10748fa227ea6ebf0730126f3f192cbdbb2c7f96a9d4685cabb86eb1cbc k8s.gcr.io/kube-scheduler-amd64:v1.24.10 registry.k8s.io/kube-scheduler-amd64:v1.24.10],SizeBytes:52751160,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.2],SizeBytes:43830714,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.2],SizeBytes:42821730,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:26802430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:245e81c2ed53b5508a6f687f3ab0c724b9ab31e6a5fa5e6d2f814981ab6ce55f registry.k8s.io/provider-aws/cloud-controller-manager:v1.24.3],SizeBytes:18282737,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.2],SizeBytes:4965809,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:45:09.965: INFO: Logging kubelet events for node i-0550ecdc8b2275df1 Jan 28 17:45:10.184: INFO: Logging pods the kubelet thinks is on node i-0550ecdc8b2275df1 Jan 28 17:45:10.420: INFO: kube-scheduler-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:10.420: INFO: Container kube-scheduler ready: true, restart count 0 Jan 28 17:45:10.420: INFO: ebs-csi-node-stxgw started at 2023-01-28 17:21:03 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:10.420: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:10.420: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:10.420: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:45:10.420: INFO: dns-controller-7d4957564c-nd8s8 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:10.420: INFO: Container dns-controller ready: true, restart count 0 Jan 28 17:45:10.420: INFO: cilium-operator-8d8478c74-2vsdk started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:10.420: INFO: Container cilium-operator ready: true, restart count 0 Jan 28 17:45:10.420: INFO: etcd-manager-events-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:10.420: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:45:10.420: INFO: etcd-manager-main-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:10.420: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:45:10.420: INFO: kube-apiserver-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+2 container statuses recorded) Jan 28 17:45:10.421: INFO: Container healthcheck ready: true, restart count 0 Jan 28 17:45:10.421: INFO: Container kube-apiserver ready: true, restart count 0 Jan 28 17:45:10.421: INFO: kube-controller-manager-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:10.421: INFO: Container kube-controller-manager ready: true, restart count 3 Jan 28 17:45:10.421: INFO: kops-controller-gll72 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:10.421: INFO: Container kops-controller ready: true, restart count 0 Jan 28 17:45:10.421: INFO: cilium-4trt7 started at 2023-01-28 17:21:03 +0000 UTC (1+1 container statuses recorded) Jan 28 17:45:10.421: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:45:10.421: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:45:10.421: INFO: aws-cloud-controller-manager-w6p6f started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:10.421: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Jan 28 17:45:10.421: INFO: ebs-csi-controller-779575c67d-kx2cv started at 2023-01-28 17:21:03 +0000 UTC (0+5 container statuses recorded) Jan 28 17:45:10.421: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:45:10.421: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:45:10.421: INFO: Container csi-resizer ready: true, restart count 0 Jan 28 17:45:10.421: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:10.421: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:11.115: INFO: Latency metrics for node i-0550ecdc8b2275df1 Jan 28 17:45:11.115: INFO: Logging node info for node i-06d4799a5db582858 Jan 28 17:45:11.331: INFO: Node Info: &Node{ObjectMeta:{i-06d4799a5db582858 aacf1bf6-d2bd-4f61-aad7-83531f5d9ba1 33004 0 2023-01-28 17:22:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-06d4799a5db582858 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-06d4799a5db582858 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.139 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06d4799a5db582858"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {e2e.test Update v1 2023-01-28 17:39:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:40:14 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 17:40:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-06d4799a5db582858,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:40:26 +0000 UTC,LastTransitionTime:2023-01-28 17:22:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.139,},NodeAddress{Type:ExternalIP,Address:18.143.137.61,},NodeAddress{Type:InternalDNS,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-143-137-61.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6f00e5a83cc422a6226665122e9d,SystemUUID:ec2e6f00-e5a8-3cc4-22a6-226665122e9d,BootID:3bfeb6d1-b807-4cfa-a0a7-954aef9e3823,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc kubernetes.io/csi/ebs.csi.aws.com^vol-0dff02acc75d59bc3],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0dff02acc75d59bc3,DevicePath:,},},Config:nil,},} Jan 28 17:45:11.332: INFO: Logging kubelet events for node i-06d4799a5db582858 Jan 28 17:45:11.552: INFO: Logging pods the kubelet thinks is on node i-06d4799a5db582858 Jan 28 17:45:11.782: INFO: up-down-3-r87lj started at 2023-01-28 17:39:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container up-down-3 ready: true, restart count 0 Jan 28 17:45:11.782: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-t9lsx started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:11.782: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-r9lbh started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:11.782: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-slm8r started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:11.782: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-zvdzg started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:11.782: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mkkk7 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:11.782: INFO: emptydir-injector started at 2023-01-28 17:40:31 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container emptydir-injector ready: false, restart count 0 Jan 28 17:45:11.782: INFO: csi-mockplugin-attacher-0 started at 2023-01-28 17:40:56 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:45:11.782: INFO: pod-configmaps-7cd33bfb-d4f5-467e-946d-cdf07018d8cf started at 2023-01-28 17:43:01 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container env-test ready: false, restart count 0 Jan 28 17:45:11.782: INFO: httpd started at 2023-01-28 17:44:35 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container httpd ready: false, restart count 0 Jan 28 17:45:11.782: INFO: up-down-2-vg8d2 started at 2023-01-28 17:38:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container up-down-2 ready: true, restart count 0 Jan 28 17:45:11.782: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-dgjbr started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:11.782: INFO: cilium-8cb5h started at 2023-01-28 17:22:14 +0000 UTC (1+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:45:11.782: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:45:11.782: INFO: ss-0 started at 2023-01-28 17:40:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container webserver ready: false, restart count 0 Jan 28 17:45:11.782: INFO: update-demo-nautilus-zqp4x started at 2023-01-28 17:44:51 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container update-demo ready: false, restart count 0 Jan 28 17:45:11.782: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-bdtqt started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:11.782: INFO: pod-server-1 started at 2023-01-28 17:40:22 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container agnhost-container ready: false, restart count 0 Jan 28 17:45:11.782: INFO: csi-mockplugin-0 started at 2023-01-28 17:40:55 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:11.782: INFO: Container csi-provisioner ready: false, restart count 0 Jan 28 17:45:11.782: INFO: Container driver-registrar ready: false, restart count 0 Jan 28 17:45:11.782: INFO: Container mock ready: false, restart count 0 Jan 28 17:45:11.782: INFO: ebs-csi-node-ttfmk started at 2023-01-28 17:22:14 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:11.782: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:11.782: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:11.782: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:45:11.782: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-lw565 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:11.782: INFO: busybox-702a2763-bdb2-44b8-ae2d-94e4cd0026af started at 2023-01-28 17:42:05 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container busybox ready: false, restart count 0 Jan 28 17:45:11.782: INFO: adopt-release-g5t8f started at 2023-01-28 17:42:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container c ready: false, restart count 0 Jan 28 17:45:11.782: INFO: rs-tbjch started at 2023-01-28 17:40:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.782: INFO: Container donothing ready: false, restart count 0 Jan 28 17:45:11.782: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-jq2xw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.783: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:11.783: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-7dx5z started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.783: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:11.783: INFO: pod-270e9e95-df4d-4169-9c6c-9329994f97a6 started at 2023-01-28 17:43:18 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:11.783: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:45:12.518: INFO: Latency metrics for node i-06d4799a5db582858 Jan 28 17:45:12.519: INFO: Logging node info for node i-0d3b431203a4a372c Jan 28 17:45:12.735: INFO: Node Info: &Node{ObjectMeta:{i-0d3b431203a4a372c b69212ba-b617-41e9-91e3-ee5929b727b2 34686 0 2023-01-28 17:22:12 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a io.kubernetes.storage.mock/node:some-mock-node kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d3b431203a4a372c kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0d3b431203a4a372c topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.33.52 csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-3106":"csi-mock-csi-mock-volumes-3106","ebs.csi.aws.com":"i-0d3b431203a4a372c"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {e2e.test Update v1 2023-01-28 17:39:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:45:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0d3b431203a4a372c,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:44:50 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:44:50 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:44:50 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:44:50 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.52,},NodeAddress{Type:ExternalIP,Address:13.229.100.227,},NodeAddress{Type:InternalDNS,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-229-100-227.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2fe0eb6b322558c1427d0b9ac3c84d,SystemUUID:ec2fe0eb-6b32-2558-c142-7d0b9ac3c84d,BootID:47c658c8-8593-40bf-a02e-622027452e1a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:45:12.736: INFO: Logging kubelet events for node i-0d3b431203a4a372c Jan 28 17:45:12.954: INFO: Logging pods the kubelet thinks is on node i-0d3b431203a4a372c Jan 28 17:45:13.178: INFO: ebs-csi-node-j2pwn started at 2023-01-28 17:22:13 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:13.178: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:13.178: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:13.178: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:45:13.178: INFO: coredns-6dd78b7459-t8lf4 started at 2023-01-28 17:22:48 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:13.178: INFO: Container coredns ready: true, restart count 0 Jan 28 17:45:13.178: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-g7jvs started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:13.178: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:13.178: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-f8tsq started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:13.178: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:13.178: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6mbxw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:13.178: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:13.178: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mmn5l started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:13.178: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:13.178: INFO: cilium-cs2hk started at 2023-01-28 17:22:13 +0000 UTC (1+1 container statuses recorded) Jan 28 17:45:13.178: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:45:13.178: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:45:13.178: INFO: csi-mockplugin-0 started at 2023-01-28 17:41:21 +0000 UTC (0+4 container statuses recorded) Jan 28 17:45:13.178: INFO: Container busybox ready: true, restart count 0 Jan 28 17:45:13.178: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:45:13.178: INFO: Container driver-registrar ready: true, restart count 0 Jan 28 17:45:13.178: INFO: Container mock ready: true, restart count 0 Jan 28 17:45:13.178: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-hnjmr started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:13.178: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:13.178: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-45pqn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:13.178: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:13.178: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-z6hs2 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:13.178: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:13.178: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-7bq4k started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:13.178: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:13.178: INFO: rs-n86z8 started at 2023-01-28 17:40:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:13.178: INFO: Container donothing ready: false, restart count 0 Jan 28 17:45:13.178: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-kmzfw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:13.178: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:13.178: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-fzmt8 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:13.178: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:13.942: INFO: Latency metrics for node i-0d3b431203a4a372c Jan 28 17:45:13.942: INFO: Logging node info for node i-0e10813507581b097 Jan 28 17:45:14.158: INFO: Node Info: &Node{ObjectMeta:{i-0e10813507581b097 8af9c758-a6f6-4a0d-b634-4d99ed71b929 32762 0 2023-01-28 17:22:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e10813507581b097 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0e10813507581b097 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.58 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e10813507581b097"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {e2e.test Update v1 2023-01-28 17:39:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:40:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0e10813507581b097,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:40:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:40:20 +0000 UTC,LastTransitionTime:2023-01-28 17:22:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.58,},NodeAddress{Type:ExternalIP,Address:54.254.187.204,},NodeAddress{Type:InternalDNS,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-187-204.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2922b3ab12fd264b764b33da3e62c2,SystemUUID:ec2922b3-ab12-fd26-4b76-4b33da3e62c2,BootID:07cb73db-252f-45aa-8a98-af3a42c0313b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:45:14.158: INFO: Logging kubelet events for node i-0e10813507581b097 Jan 28 17:45:14.376: INFO: Logging pods the kubelet thinks is on node i-0e10813507581b097 Jan 28 17:45:14.602: INFO: up-down-2-2j47t started at 2023-01-28 17:38:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container up-down-2 ready: true, restart count 0 Jan 28 17:45:14.602: INFO: ebs-csi-node-k6rdn started at 2023-01-28 17:22:08 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:14.602: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:45:14.602: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:45:14.602: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:45:14.602: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-vnxp9 started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:14.602: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-2j7n9 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:14.602: INFO: pod-736ddca1-ca58-4462-8ac4-c2357b897d98 started at 2023-01-28 17:40:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:45:14.602: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-4hpf7 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:14.602: INFO: hostexec-i-0e10813507581b097-zj6lj started at 2023-01-28 17:40:20 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:45:14.602: INFO: csi-mockplugin-0 started at 2023-01-28 17:45:01 +0000 UTC (0+3 container statuses recorded) Jan 28 17:45:14.602: INFO: Container csi-provisioner ready: false, restart count 0 Jan 28 17:45:14.602: INFO: Container driver-registrar ready: false, restart count 0 Jan 28 17:45:14.602: INFO: Container mock ready: false, restart count 0 Jan 28 17:45:14.602: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-x5gs8 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:14.602: INFO: coredns-6dd78b7459-qk7rr started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container coredns ready: true, restart count 0 Jan 28 17:45:14.602: INFO: test-ss-0 started at 2023-01-28 17:39:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container webserver ready: true, restart count 0 Jan 28 17:45:14.602: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-9hkr5 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:14.602: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-22rht started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:14.602: INFO: coredns-autoscaler-5f76bf6c7b-nnmkq started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container autoscaler ready: true, restart count 0 Jan 28 17:45:14.602: INFO: csi-mockplugin-attacher-0 started at 2023-01-28 17:45:02 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container csi-attacher ready: false, restart count 0 Jan 28 17:45:14.602: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-89jgx started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:14.602: INFO: up-down-3-bmxqn started at 2023-01-28 17:39:47 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container up-down-3 ready: false, restart count 0 Jan 28 17:45:14.602: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-rl6v2 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:45:14.602: INFO: cilium-mbfkc started at 2023-01-28 17:22:08 +0000 UTC (1+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:45:14.602: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:45:14.602: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-62lqj started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:14.602: INFO: pod-client started at 2023-01-28 17:40:19 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container pod-client ready: true, restart count 0 Jan 28 17:45:14.602: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mscfw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:45:14.602: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:45:15.354: INFO: Latency metrics for node i-0e10813507581b097 Jan 28 17:45:15.354: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "services-3650" for this suite. [AfterEach] [sig-network] Services test/e2e/network/service.go:762
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-network\]\sServices\sshould\sserve\smultiport\sendpoints\sfrom\spods\s\s\[Conformance\]$'
test/e2e/framework/framework.go:652 Jan 28 17:50:40.915: Unexpected error: <*errors.errorString | 0xc000209d30>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred test/e2e/framework/pods.go:107from junit_14.xml
[BeforeEach] [sig-network] Services test/e2e/framework/framework.go:187 �[1mSTEP�[0m: Creating a kubernetes client Jan 28 17:45:31.913: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename services �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-network] Services test/e2e/network/service.go:758 [It] should serve multiport endpoints from pods [Conformance] test/e2e/framework/framework.go:652 �[1mSTEP�[0m: creating service multi-endpoint-test in namespace services-4061 �[1mSTEP�[0m: waiting up to 3m0s for service multi-endpoint-test in namespace services-4061 to expose endpoints map[] Jan 28 17:45:34.306: INFO: successfully validated that service multi-endpoint-test in namespace services-4061 exposes endpoints map[] �[1mSTEP�[0m: Creating pod pod1 in namespace services-4061 Jan 28 17:45:34.744: INFO: The status of Pod pod1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:36.962: INFO: The status of Pod pod1 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:38.961: INFO: The status of Pod pod1 is Running (Ready = true) �[1mSTEP�[0m: waiting up to 3m0s for service multi-endpoint-test in namespace services-4061 to expose endpoints map[pod1:[100]] Jan 28 17:45:40.045: INFO: successfully validated that service multi-endpoint-test in namespace services-4061 exposes endpoints map[pod1:[100]] �[1mSTEP�[0m: Creating pod pod2 in namespace services-4061 Jan 28 17:45:40.479: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:42.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:44.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:46.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:48.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:50.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:52.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:54.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:56.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:58.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:00.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:02.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:04.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:06.698: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:08.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:10.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:12.698: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:14.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:16.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:18.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:20.695: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:22.695: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:24.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:26.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:28.695: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:30.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:32.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:34.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:36.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:38.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:40.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:42.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:44.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:46.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:48.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:50.695: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:52.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:54.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:56.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:58.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:00.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:02.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:04.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:06.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:08.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:10.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:12.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:14.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:16.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:18.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:20.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:22.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:24.695: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:26.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:28.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:30.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:32.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:34.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:36.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:38.695: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:40.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:42.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:44.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:46.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:48.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:50.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:52.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:54.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:56.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:58.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:00.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:02.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:04.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:06.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:08.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:10.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:12.698: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:14.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:16.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:18.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:20.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:22.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:24.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:26.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:28.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:30.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:32.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:34.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:36.698: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:38.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:40.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:42.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:44.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:46.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:48.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:50.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:52.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:54.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:56.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:58.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:00.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:02.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:04.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:06.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:08.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:10.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:12.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:14.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:16.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:18.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:20.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:22.698: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:24.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:26.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:28.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:30.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:32.698: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:34.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:36.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:38.698: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:40.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:42.698: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:44.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:46.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:48.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:50.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:52.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:54.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:56.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:58.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:00.699: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:02.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:04.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:06.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:08.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:10.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:12.698: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:14.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:16.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:18.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:20.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:22.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:24.698: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:26.698: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:28.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:30.695: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:32.697: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:34.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:36.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:38.696: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:40.698: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:40.914: INFO: The status of Pod pod2 is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:40.915: FAIL: Unexpected error: <*errors.errorString | 0xc000209d30>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/framework.(*PodClient).CreateSync(0xc00335b098, 0x0?) test/e2e/framework/pods.go:107 +0x94 k8s.io/kubernetes/test/e2e/network.createPodOrFail(0xd?, {0xc004a101f0, 0xd}, {0x71727c5, 0x4}, 0xc00495b980, {0xc004962bc0, 0x1, 0x1}, {0xc002ca39e0, ...}) test/e2e/network/service.go:3876 +0x255 k8s.io/kubernetes/test/e2e/network.glob..func25.5() test/e2e/network/service.go:915 +0x5de k8s.io/kubernetes/test/e2e.RunE2ETests(0x25761d7?) test/e2e/e2e.go:130 +0x6bb k8s.io/kubernetes/test/e2e.TestE2E(0x0?) test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc0001029c0, 0x741f9a8) /usr/local/go/src/testing/testing.go:1446 +0x10b created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1493 +0x35f [AfterEach] [sig-network] Services test/e2e/framework/framework.go:188 �[1mSTEP�[0m: Collecting events from namespace "services-4061". �[1mSTEP�[0m: Found 16 events. Jan 28 17:50:41.592: INFO: At 2023-01-28 17:45:34 +0000 UTC - event for pod1: {default-scheduler } Scheduled: Successfully assigned services-4061/pod1 to i-03342b7c6ebddc1c6 Jan 28 17:50:41.592: INFO: At 2023-01-28 17:45:36 +0000 UTC - event for pod1: {kubelet i-03342b7c6ebddc1c6} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:50:41.592: INFO: At 2023-01-28 17:45:36 +0000 UTC - event for pod1: {kubelet i-03342b7c6ebddc1c6} Created: Created container agnhost-container Jan 28 17:50:41.592: INFO: At 2023-01-28 17:45:36 +0000 UTC - event for pod1: {kubelet i-03342b7c6ebddc1c6} Started: Started container agnhost-container Jan 28 17:50:41.592: INFO: At 2023-01-28 17:45:40 +0000 UTC - event for pod2: {default-scheduler } Scheduled: Successfully assigned services-4061/pod2 to i-06d4799a5db582858 Jan 28 17:50:41.592: INFO: At 2023-01-28 17:45:40 +0000 UTC - event for pod2: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c9d7877cfbf02f6f8feee2453da1473865ea5d3bce4c5d7a5f3d9adc8395f429": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:41.592: INFO: At 2023-01-28 17:45:54 +0000 UTC - event for pod2: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7eaa58aea00426085e1d2b1f3ef490a09c30805f2999c46f721ccbfc653bf3fe": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:41.592: INFO: At 2023-01-28 17:46:05 +0000 UTC - event for pod2: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c8bd4667394feeeb91303306e8c6ac1a0fe0e19a14098bcf85a4373cde33d915": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:41.592: INFO: At 2023-01-28 17:46:17 +0000 UTC - event for pod2: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "08143b61deb183f6323083b413fc69270daaa976bddf77543110105888e9fdca": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:41.592: INFO: At 2023-01-28 17:46:33 +0000 UTC - event for pod2: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b4fc5dc4f4f56b2e49d03c5fd8021da3c4913c7bf7dbae1bd4ec84287dddd153": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:41.592: INFO: At 2023-01-28 17:46:46 +0000 UTC - event for pod2: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b9119ce5168fcc23336bbded58040cbcd74d25da3ec6fb64645370121e6b8aff": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:41.592: INFO: At 2023-01-28 17:47:01 +0000 UTC - event for pod2: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "812b5fc5b7f69d4d0a85034864c94581471fa58c696f76a8615abfb82fc6fb0c": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:41.592: INFO: At 2023-01-28 17:47:15 +0000 UTC - event for pod2: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "721feaf41bfea4699cb6ece54d369f967017b01d09716175ab047a9bdc5a23a2": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:41.592: INFO: At 2023-01-28 17:47:27 +0000 UTC - event for pod2: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8b539885ae823716f33826bb098314e13a8d1c63ed2e6a87eb739813cd5c3d6d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:41.592: INFO: At 2023-01-28 17:47:41 +0000 UTC - event for pod2: {kubelet i-06d4799a5db582858} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "df7694c8d57ee857d74877c06eb053d9870057af5951d1d14069303ab6d7889b": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:50:41.592: INFO: At 2023-01-28 17:50:41 +0000 UTC - event for pod1: {kubelet i-03342b7c6ebddc1c6} Killing: Stopping container agnhost-container Jan 28 17:50:41.809: INFO: POD NODE PHASE GRACE CONDITIONS Jan 28 17:50:41.810: INFO: pod2 i-06d4799a5db582858 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:45:40 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:45:40 +0000 UTC ContainersNotReady containers with unready status: [agnhost-container]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:45:40 +0000 UTC ContainersNotReady containers with unready status: [agnhost-container]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:45:40 +0000 UTC }] Jan 28 17:50:41.810: INFO: Jan 28 17:50:42.028: INFO: Unable to fetch services-4061/pod2/agnhost-container logs: the server rejected our request for an unknown reason (get pods pod2) Jan 28 17:50:42.246: INFO: Logging node info for node i-03342b7c6ebddc1c6 Jan 28 17:50:42.463: INFO: Node Info: &Node{ObjectMeta:{i-03342b7c6ebddc1c6 2b79525a-8646-4141-9334-8d4a7a592265 36017 0 2023-01-28 17:22:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03342b7c6ebddc1c6 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-03342b7c6ebddc1c6 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.39.74 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03342b7c6ebddc1c6"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {e2e.test Update v1 2023-01-28 17:39:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:47:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-03342b7c6ebddc1c6,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.74,},NodeAddress{Type:ExternalIP,Address:52.221.218.125,},NodeAddress{Type:InternalDNS,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-218-125.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bb29c4b9b2fc4087668bfcd056585,SystemUUID:ec2bb29c-4b9b-2fc4-0876-68bfcd056585,BootID:042ee65a-a182-44df-9f74-80b116af3725,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:50:42.463: INFO: Logging kubelet events for node i-03342b7c6ebddc1c6 Jan 28 17:50:42.683: INFO: Logging pods the kubelet thinks is on node i-03342b7c6ebddc1c6 Jan 28 17:50:42.910: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-9jldw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:42.911: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mkb65 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:42.911: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-fgg6g started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:42.911: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-ch9br started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:42.911: INFO: master started at 2023-01-28 17:44:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container cntr ready: true, restart count 0 Jan 28 17:50:42.911: INFO: pod-0 started at 2023-01-28 17:47:09 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container donothing ready: false, restart count 0 Jan 28 17:50:42.911: INFO: test-rollover-controller-nvj8x started at 2023-01-28 17:48:00 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container httpd ready: false, restart count 0 Jan 28 17:50:42.911: INFO: pod-8b1ca53e-c2c1-4d2c-8354-ebf444c76264 started at 2023-01-28 17:47:07 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:50:42.911: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6b7pn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:42.911: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mnncj started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:42.911: INFO: csi-hostpathplugin-0 started at 2023-01-28 17:50:24 +0000 UTC (0+7 container statuses recorded) Jan 28 17:50:42.911: INFO: Container csi-attacher ready: false, restart count 0 Jan 28 17:50:42.911: INFO: Container csi-provisioner ready: false, restart count 0 Jan 28 17:50:42.911: INFO: Container csi-resizer ready: false, restart count 0 Jan 28 17:50:42.911: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 28 17:50:42.911: INFO: Container hostpath ready: false, restart count 0 Jan 28 17:50:42.911: INFO: Container liveness-probe ready: false, restart count 0 Jan 28 17:50:42.911: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 28 17:50:42.911: INFO: test-rs-fl9dp started at 2023-01-28 17:50:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container httpd ready: false, restart count 0 Jan 28 17:50:42.911: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-kcssq started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:42.911: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-57xvw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:42.911: INFO: terminate-cmd-rpn22cf6bad-1ba4-40d5-a5a0-310f21192352 started at 2023-01-28 17:50:09 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container terminate-cmd-rpn ready: false, restart count 0 Jan 28 17:50:42.911: INFO: ebs-csi-node-xxvv5 started at 2023-01-28 17:22:12 +0000 UTC (0+3 container statuses recorded) Jan 28 17:50:42.911: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:50:42.911: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:50:42.911: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:50:42.911: INFO: affinity-nodeport-2vxfn started at 2023-01-28 17:47:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container affinity-nodeport ready: false, restart count 0 Jan 28 17:50:42.911: INFO: echo-sourceip started at 2023-01-28 17:48:29 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container agnhost-container ready: false, restart count 0 Jan 28 17:50:42.911: INFO: e2e-6bwsq-2vnvv started at 2023-01-28 17:45:45 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container c ready: false, restart count 0 Jan 28 17:50:42.911: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-jqzfn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:42.911: INFO: cilium-wlwtf started at 2023-01-28 17:22:12 +0000 UTC (1+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:50:42.911: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:50:42.911: INFO: hostexec-i-03342b7c6ebddc1c6-s4cmv started at 2023-01-28 17:44:49 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:50:42.911: INFO: slave started at 2023-01-28 17:46:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container cntr ready: false, restart count 0 Jan 28 17:50:42.911: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6vfpn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:42.911: INFO: pod-projected-configmaps-232ce7a1-e69f-4c43-88ec-9422d5087ff4 started at 2023-01-28 17:47:29 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container agnhost-container ready: false, restart count 0 Jan 28 17:50:42.911: INFO: httpd started at 2023-01-28 17:47:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:42.911: INFO: Container httpd ready: false, restart count 0 Jan 28 17:50:43.844: INFO: Latency metrics for node i-03342b7c6ebddc1c6 Jan 28 17:50:43.844: INFO: Logging node info for node i-0550ecdc8b2275df1 Jan 28 17:50:44.061: INFO: Node Info: &Node{ObjectMeta:{i-0550ecdc8b2275df1 e2f1fe4c-911f-4da0-9481-51f08c3e654e 37029 0 2023-01-28 17:20:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0550ecdc8b2275df1 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.60.182 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0550ecdc8b2275df1"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:20:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-01-28 17:20:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-01-28 17:21:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-28 17:22:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0550ecdc8b2275df1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:21:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.182,},NodeAddress{Type:ExternalIP,Address:52.221.229.242,},NodeAddress{Type:InternalDNS,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-229-242.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec22a890f5274dd8ae5c60b32c44aae9,SystemUUID:ec22a890-f527-4dd8-ae5c-60b32c44aae9,BootID:83ea7fa2-7c97-4062-a52f-a856b154a02c,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:231502799,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64@sha256:6db2761074bae8385aa601d4b45ab065893cfdc258d8b2593fc56698c44dbee5 k8s.gcr.io/kube-apiserver-amd64:v1.24.10 registry.k8s.io/kube-apiserver-amd64:v1.24.10],SizeBytes:131733971,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64@sha256:8867df47c332bae8dfd061b4d4230cd0febcc5a8a427ecb2e8f70b33ffc6d732 k8s.gcr.io/kube-controller-manager-amd64:v1.24.10 registry.k8s.io/kube-controller-manager-amd64:v1.24.10],SizeBytes:121342265,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64@sha256:a589f10748fa227ea6ebf0730126f3f192cbdbb2c7f96a9d4685cabb86eb1cbc k8s.gcr.io/kube-scheduler-amd64:v1.24.10 registry.k8s.io/kube-scheduler-amd64:v1.24.10],SizeBytes:52751160,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.2],SizeBytes:43830714,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.2],SizeBytes:42821730,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:26802430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:245e81c2ed53b5508a6f687f3ab0c724b9ab31e6a5fa5e6d2f814981ab6ce55f registry.k8s.io/provider-aws/cloud-controller-manager:v1.24.3],SizeBytes:18282737,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.2],SizeBytes:4965809,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:50:44.061: INFO: Logging kubelet events for node i-0550ecdc8b2275df1 Jan 28 17:50:44.280: INFO: Logging pods the kubelet thinks is on node i-0550ecdc8b2275df1 Jan 28 17:50:44.504: INFO: kube-scheduler-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:44.505: INFO: Container kube-scheduler ready: true, restart count 0 Jan 28 17:50:44.505: INFO: ebs-csi-node-stxgw started at 2023-01-28 17:21:03 +0000 UTC (0+3 container statuses recorded) Jan 28 17:50:44.505: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:50:44.505: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:50:44.505: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:50:44.505: INFO: dns-controller-7d4957564c-nd8s8 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:44.505: INFO: Container dns-controller ready: true, restart count 0 Jan 28 17:50:44.505: INFO: cilium-operator-8d8478c74-2vsdk started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:44.505: INFO: Container cilium-operator ready: true, restart count 0 Jan 28 17:50:44.505: INFO: etcd-manager-events-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:44.505: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:50:44.505: INFO: etcd-manager-main-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:44.505: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:50:44.505: INFO: kube-apiserver-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+2 container statuses recorded) Jan 28 17:50:44.505: INFO: Container healthcheck ready: true, restart count 0 Jan 28 17:50:44.505: INFO: Container kube-apiserver ready: true, restart count 0 Jan 28 17:50:44.505: INFO: kube-controller-manager-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:44.505: INFO: Container kube-controller-manager ready: true, restart count 3 Jan 28 17:50:44.505: INFO: kops-controller-gll72 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:44.505: INFO: Container kops-controller ready: true, restart count 0 Jan 28 17:50:44.505: INFO: cilium-4trt7 started at 2023-01-28 17:21:03 +0000 UTC (1+1 container statuses recorded) Jan 28 17:50:44.505: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:50:44.505: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:50:44.505: INFO: aws-cloud-controller-manager-w6p6f started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:44.505: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Jan 28 17:50:44.505: INFO: ebs-csi-controller-779575c67d-kx2cv started at 2023-01-28 17:21:03 +0000 UTC (0+5 container statuses recorded) Jan 28 17:50:44.505: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:50:44.505: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:50:44.505: INFO: Container csi-resizer ready: true, restart count 0 Jan 28 17:50:44.505: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:50:44.505: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:50:45.200: INFO: Latency metrics for node i-0550ecdc8b2275df1 Jan 28 17:50:45.200: INFO: Logging node info for node i-06d4799a5db582858 Jan 28 17:50:45.417: INFO: Node Info: &Node{ObjectMeta:{i-06d4799a5db582858 aacf1bf6-d2bd-4f61-aad7-83531f5d9ba1 36980 0 2023-01-28 17:22:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-06d4799a5db582858 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-06d4799a5db582858 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.139 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06d4799a5db582858"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {e2e.test Update v1 2023-01-28 17:39:16 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:40:14 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 17:40:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-06d4799a5db582858,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.139,},NodeAddress{Type:ExternalIP,Address:18.143.137.61,},NodeAddress{Type:InternalDNS,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-143-137-61.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6f00e5a83cc422a6226665122e9d,SystemUUID:ec2e6f00-e5a8-3cc4-22a6-226665122e9d,BootID:3bfeb6d1-b807-4cfa-a0a7-954aef9e3823,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:49642095,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0870183487a8091e2 kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0870183487a8091e2,DevicePath:,},},Config:nil,},} Jan 28 17:50:45.418: INFO: Logging kubelet events for node i-06d4799a5db582858 Jan 28 17:50:45.636: INFO: Logging pods the kubelet thinks is on node i-06d4799a5db582858 Jan 28 17:50:45.864: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-jq2xw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:45.864: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-7dx5z started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:45.864: INFO: pod-270e9e95-df4d-4169-9c6c-9329994f97a6 started at 2023-01-28 17:43:18 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:50:45.864: INFO: test-rs-2cpll started at 2023-01-28 17:45:37 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container httpd ready: true, restart count 0 Jan 28 17:50:45.864: INFO: pod-1 started at 2023-01-28 17:47:09 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container donothing ready: false, restart count 0 Jan 28 17:50:45.864: INFO: hostexec-i-06d4799a5db582858-4b8ml started at 2023-01-28 17:47:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:50:45.864: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-t9lsx started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:45.864: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-r9lbh started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:45.864: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-slm8r started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:45.864: INFO: pod2 started at 2023-01-28 17:45:40 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container agnhost-container ready: false, restart count 0 Jan 28 17:50:45.864: INFO: e2e-6bwsq-8fx8x started at 2023-01-28 17:45:45 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container c ready: false, restart count 0 Jan 28 17:50:45.864: INFO: pod-subpath-test-inlinevolume-r6x2 started at 2023-01-28 17:50:10 +0000 UTC (0+2 container statuses recorded) Jan 28 17:50:45.864: INFO: Container test-container-subpath-inlinevolume-r6x2 ready: false, restart count 0 Jan 28 17:50:45.864: INFO: Container test-container-volume-inlinevolume-r6x2 ready: false, restart count 0 Jan 28 17:50:45.864: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-zvdzg started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:45.864: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mkkk7 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:45.864: INFO: e2e-6bwsq-qxq2h started at 2023-01-28 17:47:56 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container c ready: false, restart count 0 Jan 28 17:50:45.864: INFO: pod-12357688-a276-4eb3-96ce-6a76a486ef13 started at 2023-01-28 17:47:18 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:50:45.864: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-dgjbr started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:45.864: INFO: cilium-8cb5h started at 2023-01-28 17:22:14 +0000 UTC (1+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:50:45.864: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:50:45.864: INFO: e2e-6bwsq-7dz2j started at 2023-01-28 17:46:22 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container c ready: false, restart count 0 Jan 28 17:50:45.864: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-bdtqt started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:45.864: INFO: affinity-nodeport-9d4b5 started at 2023-01-28 17:47:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container affinity-nodeport ready: true, restart count 0 Jan 28 17:50:45.864: INFO: local-injector started at 2023-01-28 17:47:20 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container local-injector ready: false, restart count 0 Jan 28 17:50:45.864: INFO: ebs-csi-node-ttfmk started at 2023-01-28 17:22:14 +0000 UTC (0+3 container statuses recorded) Jan 28 17:50:45.864: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:50:45.864: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:50:45.864: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:50:45.864: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-lw565 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:45.864: INFO: ephemeral-containers-target-pod started at 2023-01-28 17:47:14 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:45.864: INFO: Container test-container-1 ready: false, restart count 0 Jan 28 17:50:46.615: INFO: Latency metrics for node i-06d4799a5db582858 Jan 28 17:50:46.615: INFO: Logging node info for node i-0d3b431203a4a372c Jan 28 17:50:46.831: INFO: Node Info: &Node{ObjectMeta:{i-0d3b431203a4a372c b69212ba-b617-41e9-91e3-ee5929b727b2 37855 0 2023-01-28 17:22:12 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a io.kubernetes.storage.mock/node:some-mock-node kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d3b431203a4a372c kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0d3b431203a4a372c topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.33.52 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d3b431203a4a372c"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {e2e.test Update v1 2023-01-28 17:39:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:45:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0d3b431203a4a372c,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:28 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:28 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:28 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:50:28 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.52,},NodeAddress{Type:ExternalIP,Address:13.229.100.227,},NodeAddress{Type:InternalDNS,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-229-100-227.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2fe0eb6b322558c1427d0b9ac3c84d,SystemUUID:ec2fe0eb-6b32-2558-c142-7d0b9ac3c84d,BootID:47c658c8-8593-40bf-a02e-622027452e1a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:50:46.832: INFO: Logging kubelet events for node i-0d3b431203a4a372c Jan 28 17:50:47.050: INFO: Logging pods the kubelet thinks is on node i-0d3b431203a4a372c Jan 28 17:50:47.282: INFO: coredns-6dd78b7459-t8lf4 started at 2023-01-28 17:22:48 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:47.282: INFO: Container coredns ready: true, restart count 0 Jan 28 17:50:47.282: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-g7jvs started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:47.282: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:47.282: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-f8tsq started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:47.282: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:47.282: INFO: hostexec-i-0d3b431203a4a372c-5n26x started at 2023-01-28 17:50:27 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:47.282: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:50:47.282: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-6mbxw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:47.282: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:47.282: INFO: ebs-csi-node-j2pwn started at 2023-01-28 17:22:13 +0000 UTC (0+3 container statuses recorded) Jan 28 17:50:47.282: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:50:47.282: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:50:47.282: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:50:47.282: INFO: pod-2934e8e9-9ba5-4f70-a4c1-3be753776a5f started at 2023-01-28 17:50:33 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:47.282: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:50:47.282: INFO: csi-mockplugin-0 started at 2023-01-28 17:41:21 +0000 UTC (0+4 container statuses recorded) Jan 28 17:50:47.282: INFO: Container busybox ready: true, restart count 0 Jan 28 17:50:47.282: INFO: Container csi-provisioner ready: true, restart count 2 Jan 28 17:50:47.282: INFO: Container driver-registrar ready: true, restart count 0 Jan 28 17:50:47.282: INFO: Container mock ready: true, restart count 0 Jan 28 17:50:47.282: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-hnjmr started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:47.282: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:47.282: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mmn5l started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:47.282: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:47.282: INFO: cilium-cs2hk started at 2023-01-28 17:22:13 +0000 UTC (1+1 container statuses recorded) Jan 28 17:50:47.282: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:50:47.282: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:50:47.282: INFO: csi-mockplugin-0 started at 2023-01-28 17:47:23 +0000 UTC (0+3 container statuses recorded) Jan 28 17:50:47.282: INFO: Container csi-provisioner ready: false, restart count 0 Jan 28 17:50:47.282: INFO: Container driver-registrar ready: false, restart count 0 Jan 28 17:50:47.282: INFO: Container mock ready: false, restart count 0 Jan 28 17:50:47.282: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-45pqn started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:47.282: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:47.282: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-z6hs2 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:47.282: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:47.282: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-7bq4k started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:47.282: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:47.282: INFO: test-rs-sr5rf started at 2023-01-28 17:50:24 +0000 UTC (0+2 container statuses recorded) Jan 28 17:50:47.282: INFO: Container httpd ready: false, restart count 0 Jan 28 17:50:47.282: INFO: Container test-rs ready: false, restart count 0 Jan 28 17:50:47.282: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-kmzfw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:47.282: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:47.282: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-fzmt8 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:47.282: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:47.282: INFO: csi-mockplugin-resizer-0 started at 2023-01-28 17:47:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:47.282: INFO: Container csi-resizer ready: false, restart count 0 Jan 28 17:50:48.065: INFO: Latency metrics for node i-0d3b431203a4a372c Jan 28 17:50:48.066: INFO: Logging node info for node i-0e10813507581b097 Jan 28 17:50:48.282: INFO: Node Info: &Node{ObjectMeta:{i-0e10813507581b097 8af9c758-a6f6-4a0d-b634-4d99ed71b929 37803 0 2023-01-28 17:22:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e10813507581b097 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0e10813507581b097 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.58 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e10813507581b097"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {e2e.test Update v1 2023-01-28 17:39:15 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2023-01-28 17:40:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0e10813507581b097,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:34 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:34 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:34 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:50:34 +0000 UTC,LastTransitionTime:2023-01-28 17:22:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.58,},NodeAddress{Type:ExternalIP,Address:54.254.187.204,},NodeAddress{Type:InternalDNS,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-187-204.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2922b3ab12fd264b764b33da3e62c2,SystemUUID:ec2922b3-ab12-fd26-4b76-4b33da3e62c2,BootID:07cb73db-252f-45aa-8a98-af3a42c0313b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:50:48.283: INFO: Logging kubelet events for node i-0e10813507581b097 Jan 28 17:50:48.502: INFO: Logging pods the kubelet thinks is on node i-0e10813507581b097 Jan 28 17:50:48.731: INFO: pod-0e370f03-c117-4110-99af-41ac7db026f8 started at 2023-01-28 17:46:04 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container write-pod ready: false, restart count 0 Jan 28 17:50:48.731: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-9hkr5 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:48.731: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-22rht started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:48.731: INFO: coredns-6dd78b7459-qk7rr started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container coredns ready: true, restart count 0 Jan 28 17:50:48.731: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-89jgx started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:48.731: INFO: coredns-autoscaler-5f76bf6c7b-nnmkq started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container autoscaler ready: true, restart count 0 Jan 28 17:50:48.731: INFO: csi-mockplugin-attacher-0 started at 2023-01-28 17:45:02 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:50:48.731: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-rl6v2 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:48.731: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-mscfw started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:48.731: INFO: cilium-mbfkc started at 2023-01-28 17:22:08 +0000 UTC (1+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:50:48.731: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:50:48.731: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-62lqj started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: false, restart count 0 Jan 28 17:50:48.731: INFO: boom-server started at 2023-01-28 17:48:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container boom-server ready: false, restart count 0 Jan 28 17:50:48.731: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-2j7n9 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:48.731: INFO: ebs-csi-node-k6rdn started at 2023-01-28 17:22:08 +0000 UTC (0+3 container statuses recorded) Jan 28 17:50:48.731: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:50:48.731: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:50:48.731: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:50:48.731: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-vnxp9 started at 2023-01-28 17:39:16 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:48.731: INFO: csi-mockplugin-0 started at 2023-01-28 17:45:01 +0000 UTC (0+3 container statuses recorded) Jan 28 17:50:48.731: INFO: Container csi-provisioner ready: false, restart count 0 Jan 28 17:50:48.731: INFO: Container driver-registrar ready: false, restart count 0 Jan 28 17:50:48.731: INFO: Container mock ready: false, restart count 0 Jan 28 17:50:48.731: INFO: affinity-nodeport-2mdr9 started at 2023-01-28 17:47:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container affinity-nodeport ready: false, restart count 0 Jan 28 17:50:48.731: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-4hpf7 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:48.731: INFO: hostexec-i-0e10813507581b097-7x65b started at 2023-01-28 17:45:59 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:50:48.731: INFO: cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f-x5gs8 started at 2023-01-28 17:39:17 +0000 UTC (0+1 container statuses recorded) Jan 28 17:50:48.731: INFO: Container cleanup40-33bddc3f-730e-48e4-b4e9-51928a8f8e7f ready: true, restart count 0 Jan 28 17:50:49.471: INFO: Latency metrics for node i-0e10813507581b097 Jan 28 17:50:49.471: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "services-4061" for this suite. [AfterEach] [sig-network] Services test/e2e/network/service.go:762
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-node\]\sMount\spropagation\sshould\spropagate\smounts\swithin\sdefined\sscopes$'
test/e2e/node/mount_propagation.go:85 Jan 28 17:51:25.693: Unexpected error: <*errors.errorString | 0xc0001f21b0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred test/e2e/framework/pods.go:107from junit_20.xml
[BeforeEach] [sig-node] Mount propagation test/e2e/framework/framework.go:187 �[1mSTEP�[0m: Creating a kubernetes client Jan 28 17:44:32.255: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename mount-propagation �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should propagate mounts within defined scopes test/e2e/node/mount_propagation.go:85 Jan 28 17:44:34.404: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:36.618: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:38.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:40.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:42.621: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:44.620: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:46.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:48.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:50.621: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:52.618: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:54.618: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:56.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:44:58.620: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:00.622: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:02.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:04.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:06.618: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:08.618: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:10.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:12.620: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:14.620: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:16.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:18.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:20.621: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:22.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:24.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:26.618: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:28.620: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:30.620: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:32.622: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:34.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:36.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:38.620: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:40.620: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:42.620: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:44.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:46.618: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:48.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:50.620: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:52.620: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:54.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:56.618: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:45:58.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:00.618: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:02.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:04.618: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:06.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:08.618: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:10.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:12.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:14.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:16.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:18.618: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:20.618: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:22.619: INFO: The status of Pod master is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:24.619: INFO: The status of Pod master is Running (Ready = true) Jan 28 17:46:25.263: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:27.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:29.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:31.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:33.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:35.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:37.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:39.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:41.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:43.480: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:45.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:47.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:49.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:51.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:53.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:55.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:57.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:46:59.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:01.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:03.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:05.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:07.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:09.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:11.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:13.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:15.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:17.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:19.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:21.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:23.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:25.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:27.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:29.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:31.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:33.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:35.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:37.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:39.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:41.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:43.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:45.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:47.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:49.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:51.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:53.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:55.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:57.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:47:59.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:01.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:03.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:05.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:07.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:09.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:11.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:13.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:15.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:17.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:19.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:21.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:23.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:25.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:27.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:29.480: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:31.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:33.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:35.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:37.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:39.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:41.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:43.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:45.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:47.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:49.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:51.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:53.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:55.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:57.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:48:59.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:01.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:03.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:05.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:07.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:09.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:11.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:13.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:15.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:17.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:19.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:21.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:23.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:25.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:27.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:29.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:31.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:33.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:35.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:37.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:39.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:41.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:43.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:45.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:47.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:49.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:51.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:53.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:55.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:57.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:49:59.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:01.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:03.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:05.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:07.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:09.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:11.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:13.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:15.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:17.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:19.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:21.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:23.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:25.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:27.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:29.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:31.481: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:33.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:35.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:37.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:39.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:41.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:43.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:45.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:47.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:49.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:51.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:53.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:55.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:57.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:50:59.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:51:01.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:51:03.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:51:05.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:51:07.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:51:09.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:51:11.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:51:13.477: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:51:15.479: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:51:17.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:51:19.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:51:21.481: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:51:23.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:51:25.478: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:51:25.692: INFO: The status of Pod slave is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:51:25.693: FAIL: Unexpected error: <*errors.errorString | 0xc0001f21b0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/framework.(*PodClient).CreateSync(0xc0028c36f8, 0x5?) test/e2e/framework/pods.go:107 +0x94 k8s.io/kubernetes/test/e2e/node.glob..func7.1() test/e2e/node/mount_propagation.go:119 +0x2b9 k8s.io/kubernetes/test/e2e.RunE2ETests(0x25761d7?) test/e2e/e2e.go:130 +0x6bb k8s.io/kubernetes/test/e2e.TestE2E(0x24e52d9?) test/e2e/e2e_test.go:136 +0x19 testing.tRunner(0xc000c93ba0, 0x741f9a8) /usr/local/go/src/testing/testing.go:1446 +0x10b created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1493 +0x35f Jan 28 17:51:30.343: INFO: ExecWithOptions {Command:[nsenter --mount=/rootfs/proc/1/ns/mnt -- sh -c rm -rf "/var/lib/kubelet/mount-propagation-9877"] Namespace:mount-propagation-9877 PodName:hostexec-i-03342b7c6ebddc1c6-xmws5 ContainerName:agnhost-container Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:true Quiet:false} Jan 28 17:51:30.343: INFO: >>> kubeConfig: /root/.kube/config Jan 28 17:51:30.344: INFO: ExecWithOptions: Clientset creation Jan 28 17:51:30.344: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-grid-cilium-eni-deb10-k24-ko26.test-cncf-aws.k8s.io/api/v1/namespaces/mount-propagation-9877/pods/hostexec-i-03342b7c6ebddc1c6-xmws5/exec?command=nsenter&command=--mount%3D%2Frootfs%2Fproc%2F1%2Fns%2Fmnt&command=--&command=sh&command=-c&command=rm+-rf+%22%2Fvar%2Flib%2Fkubelet%2Fmount-propagation-9877%22&container=agnhost-container&container=agnhost-container&stderr=true&stdout=true) �[1mSTEP�[0m: Deleting pod hostexec-i-03342b7c6ebddc1c6-xmws5 in namespace mount-propagation-9877 [AfterEach] [sig-node] Mount propagation test/e2e/framework/framework.go:188 �[1mSTEP�[0m: Collecting events from namespace "mount-propagation-9877". �[1mSTEP�[0m: Found 26 events. Jan 28 17:51:32.241: INFO: At 2023-01-28 17:44:34 +0000 UTC - event for master: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ca330c438c8c9b0f2a9e561bf5faddcc58549bfc9a78d0a1048c5bd12e6cc296": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:44:47 +0000 UTC - event for master: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2894a005a76dd551cce317de7ee2e848f0c5f9cb225e6f11ea5faab86016cc04": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:44:59 +0000 UTC - event for master: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "187f3b9fbdf984381f2b60c781ac6a4873362b7be935dc9be5f1b3283c86ac4a": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:45:12 +0000 UTC - event for master: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "61a7a0fbfc700127ec5b3b85e4ca18c1525e4f0497ff49d9eb63ff9f8d7c8550": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:45:27 +0000 UTC - event for master: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8727ec75357782ea21fd3faca01789a25541558d8a43ebf8db5973e8b8deead3": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:45:39 +0000 UTC - event for master: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "badbb928e5ec7403974603d4b0e3d2649a569423ac42af489d31b08248eeca29": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:45:55 +0000 UTC - event for master: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6311a4c7046bd41437e9a33dc6894b982b9953575e3019cc5fff3e0658714daf": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:46:07 +0000 UTC - event for master: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f3577bc0385b18085262565557fb94bca83e8b3a2cb33fb2596241c7bdd845a0": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:46:23 +0000 UTC - event for master: {kubelet i-03342b7c6ebddc1c6} Started: Started container cntr Jan 28 17:51:32.242: INFO: At 2023-01-28 17:46:23 +0000 UTC - event for master: {kubelet i-03342b7c6ebddc1c6} Pulled: Container image "k8s.gcr.io/e2e-test-images/busybox:1.29-2" already present on machine Jan 28 17:51:32.242: INFO: At 2023-01-28 17:46:23 +0000 UTC - event for master: {kubelet i-03342b7c6ebddc1c6} Created: Created container cntr Jan 28 17:51:32.242: INFO: At 2023-01-28 17:46:25 +0000 UTC - event for slave: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5ea6699183ace68c0780d25e274a26624f6be640d074969b61f58036ec83ead6": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:46:38 +0000 UTC - event for slave: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9263c04c49c04d07dce624791e946f44cc7be71d70f483ed7f5ace92a69f15d7": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:46:49 +0000 UTC - event for slave: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fe112ffe1527a6f7949ab4cc38092fcb7aba044c1615388d2b158a140eac023d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:47:03 +0000 UTC - event for slave: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0b3322cce9978fe5764f47e7d6f1aba621fdedc78d5bc94d1868eb64ad3f631d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:47:19 +0000 UTC - event for slave: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7901c62ceea4940e9d39f37e8032e86e5aefeaa04ec10d4c3987a0bc672d9ec8": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:47:30 +0000 UTC - event for slave: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "be621162fe51bb3747397ec5ed55a15197284061185158444de3fa769d573833": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:47:42 +0000 UTC - event for slave: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8e42a3ccc66307a077c0af6d73a30b05c31c1e98035cdfd1b684e2188e3d4c2d": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:47:55 +0000 UTC - event for slave: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0b04b4c72ea94dc2b8878ac571b7435e4d449725746451d1b6a805b5b57ff1b1": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:48:10 +0000 UTC - event for slave: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e43829d78bf19cbb37c05b71964f13b020c5ebf09a5eb2bdf5926de2e3118639": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:48:24 +0000 UTC - event for slave: {kubelet i-03342b7c6ebddc1c6} FailedCreatePodSandBox: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7aebcbdefc4da38e7ed6a864e51e18bc7c2e20ebeb96301e306e25363ce8d8d1": plugin type="cilium-cni" name="cilium" failed (add): unable to allocate IP via local cilium agent: [POST /ipam][502] postIpamFailure No more IPs available Jan 28 17:51:32.242: INFO: At 2023-01-28 17:51:25 +0000 UTC - event for hostexec-i-03342b7c6ebddc1c6-xmws5: {default-scheduler } Scheduled: Successfully assigned mount-propagation-9877/hostexec-i-03342b7c6ebddc1c6-xmws5 to i-03342b7c6ebddc1c6 Jan 28 17:51:32.242: INFO: At 2023-01-28 17:51:26 +0000 UTC - event for hostexec-i-03342b7c6ebddc1c6-xmws5: {kubelet i-03342b7c6ebddc1c6} Pulled: Container image "k8s.gcr.io/e2e-test-images/agnhost:2.39" already present on machine Jan 28 17:51:32.242: INFO: At 2023-01-28 17:51:26 +0000 UTC - event for hostexec-i-03342b7c6ebddc1c6-xmws5: {kubelet i-03342b7c6ebddc1c6} Created: Created container agnhost-container Jan 28 17:51:32.242: INFO: At 2023-01-28 17:51:26 +0000 UTC - event for hostexec-i-03342b7c6ebddc1c6-xmws5: {kubelet i-03342b7c6ebddc1c6} Started: Started container agnhost-container Jan 28 17:51:32.242: INFO: At 2023-01-28 17:51:31 +0000 UTC - event for hostexec-i-03342b7c6ebddc1c6-xmws5: {kubelet i-03342b7c6ebddc1c6} Killing: Stopping container agnhost-container Jan 28 17:51:32.456: INFO: POD NODE PHASE GRACE CONDITIONS Jan 28 17:51:32.456: INFO: master i-03342b7c6ebddc1c6 Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:44:34 +0000 UTC } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:46:24 +0000 UTC } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:46:24 +0000 UTC } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:44:34 +0000 UTC }] Jan 28 17:51:32.456: INFO: slave i-03342b7c6ebddc1c6 Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:46:24 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:46:24 +0000 UTC ContainersNotReady containers with unready status: [cntr]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:46:24 +0000 UTC ContainersNotReady containers with unready status: [cntr]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 17:46:24 +0000 UTC }] Jan 28 17:51:32.456: INFO: Jan 28 17:51:32.897: INFO: Unable to fetch mount-propagation-9877/slave/cntr logs: the server rejected our request for an unknown reason (get pods slave) Jan 28 17:51:33.113: INFO: Logging node info for node i-03342b7c6ebddc1c6 Jan 28 17:51:33.327: INFO: Node Info: &Node{ObjectMeta:{i-03342b7c6ebddc1c6 2b79525a-8646-4141-9334-8d4a7a592265 38553 0 2023-01-28 17:22:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-03342b7c6ebddc1c6 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-03342b7c6ebddc1c6 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.39.74 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03342b7c6ebddc1c6"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-01-28 17:47:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-03342b7c6ebddc1c6,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:47:01 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.74,},NodeAddress{Type:ExternalIP,Address:52.221.218.125,},NodeAddress{Type:InternalDNS,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-03342b7c6ebddc1c6.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-218-125.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bb29c4b9b2fc4087668bfcd056585,SystemUUID:ec2bb29c-4b9b-2fc4-0876-68bfcd056585,BootID:042ee65a-a182-44df-9f74-80b116af3725,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:51:33.328: INFO: Logging kubelet events for node i-03342b7c6ebddc1c6 Jan 28 17:51:33.548: INFO: Logging pods the kubelet thinks is on node i-03342b7c6ebddc1c6 Jan 28 17:51:33.771: INFO: csi-hostpathplugin-0 started at 2023-01-28 17:50:24 +0000 UTC (0+7 container statuses recorded) Jan 28 17:51:33.771: INFO: Container csi-attacher ready: false, restart count 0 Jan 28 17:51:33.771: INFO: Container csi-provisioner ready: false, restart count 0 Jan 28 17:51:33.771: INFO: Container csi-resizer ready: false, restart count 0 Jan 28 17:51:33.771: INFO: Container csi-snapshotter ready: false, restart count 0 Jan 28 17:51:33.771: INFO: Container hostpath ready: false, restart count 0 Jan 28 17:51:33.771: INFO: Container liveness-probe ready: false, restart count 0 Jan 28 17:51:33.771: INFO: Container node-driver-registrar ready: false, restart count 0 Jan 28 17:51:33.771: INFO: test-rs-fl9dp started at 2023-01-28 17:50:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:33.771: INFO: Container httpd ready: true, restart count 0 Jan 28 17:51:33.771: INFO: terminate-cmd-rpn22cf6bad-1ba4-40d5-a5a0-310f21192352 started at 2023-01-28 17:50:09 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:33.771: INFO: Container terminate-cmd-rpn ready: false, restart count 0 Jan 28 17:51:33.771: INFO: ebs-csi-node-xxvv5 started at 2023-01-28 17:22:12 +0000 UTC (0+3 container statuses recorded) Jan 28 17:51:33.771: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:51:33.771: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:51:33.771: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:51:33.771: INFO: affinity-nodeport-2vxfn started at 2023-01-28 17:47:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:33.771: INFO: Container affinity-nodeport ready: false, restart count 0 Jan 28 17:51:33.771: INFO: echo-sourceip started at 2023-01-28 17:48:29 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:33.771: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:51:33.771: INFO: e2e-6bwsq-2vnvv started at 2023-01-28 17:45:45 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:33.771: INFO: Container c ready: false, restart count 0 Jan 28 17:51:33.771: INFO: httpd started at 2023-01-28 17:47:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:33.771: INFO: Container httpd ready: false, restart count 0 Jan 28 17:51:33.771: INFO: cilium-wlwtf started at 2023-01-28 17:22:12 +0000 UTC (1+1 container statuses recorded) Jan 28 17:51:33.771: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:51:33.771: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:51:33.771: INFO: slave started at 2023-01-28 17:46:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:33.771: INFO: Container cntr ready: false, restart count 0 Jan 28 17:51:33.771: INFO: pod-projected-configmaps-232ce7a1-e69f-4c43-88ec-9422d5087ff4 started at 2023-01-28 17:47:29 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:33.771: INFO: Container agnhost-container ready: false, restart count 0 Jan 28 17:51:33.771: INFO: netserver-0 started at 2023-01-28 17:50:55 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:33.771: INFO: Container webserver ready: false, restart count 0 Jan 28 17:51:33.771: INFO: master started at 2023-01-28 17:44:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:33.771: INFO: Container cntr ready: true, restart count 0 Jan 28 17:51:33.771: INFO: pod-0 started at 2023-01-28 17:47:09 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:33.771: INFO: Container donothing ready: true, restart count 0 Jan 28 17:51:33.771: INFO: test-rollover-controller-nvj8x started at 2023-01-28 17:48:00 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:33.771: INFO: Container httpd ready: true, restart count 0 Jan 28 17:51:33.771: INFO: test-rollover-deployment-779c67f4f8-dgqvh started at 2023-01-28 17:50:55 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:33.771: INFO: Container agnhost ready: true, restart count 0 Jan 28 17:51:34.830: INFO: Latency metrics for node i-03342b7c6ebddc1c6 Jan 28 17:51:34.830: INFO: Logging node info for node i-0550ecdc8b2275df1 Jan 28 17:51:35.044: INFO: Node Info: &Node{ObjectMeta:{i-0550ecdc8b2275df1 e2f1fe4c-911f-4da0-9481-51f08c3e654e 37029 0 2023-01-28 17:20:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0550ecdc8b2275df1 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.60.182 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0550ecdc8b2275df1"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:20:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-01-28 17:20:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:21:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-01-28 17:21:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-28 17:22:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0550ecdc8b2275df1,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3918811136 0} {<nil>} 3826964Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3813953536 0} {<nil>} 3724564Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:20:20 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:48:24 +0000 UTC,LastTransitionTime:2023-01-28 17:21:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.182,},NodeAddress{Type:ExternalIP,Address:52.221.229.242,},NodeAddress{Type:InternalDNS,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0550ecdc8b2275df1.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-229-242.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec22a890f5274dd8ae5c60b32c44aae9,SystemUUID:ec22a890-f527-4dd8-ae5c-60b32c44aae9,BootID:83ea7fa2-7c97-4062-a52f-a856b154a02c,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:231502799,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64@sha256:6db2761074bae8385aa601d4b45ab065893cfdc258d8b2593fc56698c44dbee5 k8s.gcr.io/kube-apiserver-amd64:v1.24.10 registry.k8s.io/kube-apiserver-amd64:v1.24.10],SizeBytes:131733971,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64@sha256:8867df47c332bae8dfd061b4d4230cd0febcc5a8a427ecb2e8f70b33ffc6d732 k8s.gcr.io/kube-controller-manager-amd64:v1.24.10 registry.k8s.io/kube-controller-manager-amd64:v1.24.10],SizeBytes:121342265,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64@sha256:a589f10748fa227ea6ebf0730126f3f192cbdbb2c7f96a9d4685cabb86eb1cbc k8s.gcr.io/kube-scheduler-amd64:v1.24.10 registry.k8s.io/kube-scheduler-amd64:v1.24.10],SizeBytes:52751160,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.2],SizeBytes:43830714,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.2],SizeBytes:42821730,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:26802430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:245e81c2ed53b5508a6f687f3ab0c724b9ab31e6a5fa5e6d2f814981ab6ce55f registry.k8s.io/provider-aws/cloud-controller-manager:v1.24.3],SizeBytes:18282737,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.2],SizeBytes:4965809,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:51:35.044: INFO: Logging kubelet events for node i-0550ecdc8b2275df1 Jan 28 17:51:35.260: INFO: Logging pods the kubelet thinks is on node i-0550ecdc8b2275df1 Jan 28 17:51:35.481: INFO: dns-controller-7d4957564c-nd8s8 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:35.481: INFO: Container dns-controller ready: true, restart count 0 Jan 28 17:51:35.481: INFO: kube-scheduler-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:35.481: INFO: Container kube-scheduler ready: true, restart count 0 Jan 28 17:51:35.481: INFO: ebs-csi-node-stxgw started at 2023-01-28 17:21:03 +0000 UTC (0+3 container statuses recorded) Jan 28 17:51:35.481: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:51:35.481: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:51:35.481: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:51:35.481: INFO: kube-apiserver-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+2 container statuses recorded) Jan 28 17:51:35.481: INFO: Container healthcheck ready: true, restart count 0 Jan 28 17:51:35.481: INFO: Container kube-apiserver ready: true, restart count 0 Jan 28 17:51:35.481: INFO: kube-controller-manager-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:35.481: INFO: Container kube-controller-manager ready: true, restart count 3 Jan 28 17:51:35.481: INFO: kops-controller-gll72 started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:35.481: INFO: Container kops-controller ready: true, restart count 0 Jan 28 17:51:35.481: INFO: cilium-4trt7 started at 2023-01-28 17:21:03 +0000 UTC (1+1 container statuses recorded) Jan 28 17:51:35.481: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:51:35.481: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:51:35.481: INFO: aws-cloud-controller-manager-w6p6f started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:35.481: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Jan 28 17:51:35.481: INFO: cilium-operator-8d8478c74-2vsdk started at 2023-01-28 17:21:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:35.481: INFO: Container cilium-operator ready: true, restart count 0 Jan 28 17:51:35.481: INFO: etcd-manager-events-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:35.481: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:51:35.481: INFO: etcd-manager-main-i-0550ecdc8b2275df1 started at 2023-01-28 17:19:44 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:35.481: INFO: Container etcd-manager ready: true, restart count 0 Jan 28 17:51:35.481: INFO: ebs-csi-controller-779575c67d-kx2cv started at 2023-01-28 17:21:03 +0000 UTC (0+5 container statuses recorded) Jan 28 17:51:35.481: INFO: Container csi-attacher ready: true, restart count 0 Jan 28 17:51:35.481: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:51:35.481: INFO: Container csi-resizer ready: true, restart count 0 Jan 28 17:51:35.481: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:51:35.481: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:51:36.205: INFO: Latency metrics for node i-0550ecdc8b2275df1 Jan 28 17:51:36.205: INFO: Logging node info for node i-06d4799a5db582858 Jan 28 17:51:36.419: INFO: Node Info: &Node{ObjectMeta:{i-06d4799a5db582858 aacf1bf6-d2bd-4f61-aad7-83531f5d9ba1 38577 0 2023-01-28 17:22:13 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-06d4799a5db582858 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-06d4799a5db582858 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.139 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-06d4799a5db582858"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:40:14 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 17:40:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-06d4799a5db582858,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:13 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:48:14 +0000 UTC,LastTransitionTime:2023-01-28 17:22:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.139,},NodeAddress{Type:ExternalIP,Address:18.143.137.61,},NodeAddress{Type:InternalDNS,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-06d4799a5db582858.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-18-143-137-61.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2e6f00e5a83cc422a6226665122e9d,SystemUUID:ec2e6f00-e5a8-3cc4-22a6-226665122e9d,BootID:3bfeb6d1-b807-4cfa-a0a7-954aef9e3823,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c k8s.gcr.io/e2e-test-images/nautilus:1.5],SizeBytes:49642095,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0870183487a8091e2 kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0a060c54964200bcc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0870183487a8091e2,DevicePath:,},},Config:nil,},} Jan 28 17:51:36.420: INFO: Logging kubelet events for node i-06d4799a5db582858 Jan 28 17:51:36.636: INFO: Logging pods the kubelet thinks is on node i-06d4799a5db582858 Jan 28 17:51:36.866: INFO: e2e-6bwsq-qxq2h started at 2023-01-28 17:47:56 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:36.866: INFO: Container c ready: false, restart count 0 Jan 28 17:51:36.866: INFO: deployment-8f95b86b8-gc2h9 started at 2023-01-28 17:51:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:36.866: INFO: Container nginx ready: false, restart count 0 Jan 28 17:51:36.866: INFO: pod-12357688-a276-4eb3-96ce-6a76a486ef13 started at 2023-01-28 17:47:18 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:36.866: INFO: Container write-pod ready: true, restart count 0 Jan 28 17:51:36.866: INFO: cilium-8cb5h started at 2023-01-28 17:22:14 +0000 UTC (1+1 container statuses recorded) Jan 28 17:51:36.866: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:51:36.866: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:51:36.866: INFO: hostexec-i-06d4799a5db582858-wm7qp started at 2023-01-28 17:51:32 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:36.866: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:51:36.866: INFO: e2e-6bwsq-7dz2j started at 2023-01-28 17:46:22 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:36.866: INFO: Container c ready: false, restart count 0 Jan 28 17:51:36.866: INFO: affinity-nodeport-9d4b5 started at 2023-01-28 17:47:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:36.866: INFO: Container affinity-nodeport ready: true, restart count 0 Jan 28 17:51:36.866: INFO: local-injector started at 2023-01-28 17:47:20 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:36.866: INFO: Container local-injector ready: true, restart count 0 Jan 28 17:51:36.866: INFO: ebs-csi-node-ttfmk started at 2023-01-28 17:22:14 +0000 UTC (0+3 container statuses recorded) Jan 28 17:51:36.866: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:51:36.866: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:51:36.866: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:51:36.866: INFO: ephemeral-containers-target-pod started at 2023-01-28 17:47:14 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:36.866: INFO: Container test-container-1 ready: true, restart count 0 Jan 28 17:51:36.866: INFO: netserver-1 started at 2023-01-28 17:50:55 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:36.866: INFO: Container webserver ready: false, restart count 0 Jan 28 17:51:36.866: INFO: pod-270e9e95-df4d-4169-9c6c-9329994f97a6 started at 2023-01-28 17:43:18 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:36.866: INFO: Container write-pod ready: true, restart count 0 Jan 28 17:51:36.866: INFO: test-rs-2cpll started at 2023-01-28 17:45:37 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:36.866: INFO: Container httpd ready: true, restart count 0 Jan 28 17:51:36.866: INFO: pod-1 started at 2023-01-28 17:47:09 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:36.866: INFO: Container donothing ready: true, restart count 0 Jan 28 17:51:36.866: INFO: e2e-6bwsq-8fx8x started at 2023-01-28 17:45:45 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:36.866: INFO: Container c ready: false, restart count 0 Jan 28 17:51:36.866: INFO: hostexec-i-06d4799a5db582858-4b8ml started at 2023-01-28 17:47:03 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:36.866: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:51:36.866: INFO: pod-subpath-test-inlinevolume-r6x2 started at 2023-01-28 17:50:10 +0000 UTC (0+2 container statuses recorded) Jan 28 17:51:36.866: INFO: Container test-container-subpath-inlinevolume-r6x2 ready: false, restart count 0 Jan 28 17:51:36.866: INFO: Container test-container-volume-inlinevolume-r6x2 ready: true, restart count 0 Jan 28 17:51:37.619: INFO: Latency metrics for node i-06d4799a5db582858 Jan 28 17:51:37.619: INFO: Logging node info for node i-0d3b431203a4a372c Jan 28 17:51:37.839: INFO: Node Info: &Node{ObjectMeta:{i-0d3b431203a4a372c b69212ba-b617-41e9-91e3-ee5929b727b2 38590 0 2023-01-28 17:22:12 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d3b431203a4a372c kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0d3b431203a4a372c topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.33.52 csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-151":"i-0d3b431203a4a372c","ebs.csi.aws.com":"i-0d3b431203a4a372c"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:22:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-01-28 17:45:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0d3b431203a4a372c,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:28 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:28 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:28 +0000 UTC,LastTransitionTime:2023-01-28 17:22:12 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:50:28 +0000 UTC,LastTransitionTime:2023-01-28 17:22:32 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.33.52,},NodeAddress{Type:ExternalIP,Address:13.229.100.227,},NodeAddress{Type:InternalDNS,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d3b431203a4a372c.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-229-100-227.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2fe0eb6b322558c1427d0b9ac3c84d,SystemUUID:ec2fe0eb-6b32-2558-c142-7d0b9ac3c84d,BootID:47c658c8-8593-40bf-a02e-622027452e1a,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c k8s.gcr.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 k8s.gcr.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f k8s.gcr.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:51:37.839: INFO: Logging kubelet events for node i-0d3b431203a4a372c Jan 28 17:51:38.060: INFO: Logging pods the kubelet thinks is on node i-0d3b431203a4a372c Jan 28 17:51:38.283: INFO: pvc-volume-tester-9brq4 started at 2023-01-28 17:51:14 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:38.283: INFO: Container volume-tester ready: true, restart count 0 Jan 28 17:51:38.283: INFO: csi-mockplugin-resizer-0 started at 2023-01-28 17:47:24 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:38.283: INFO: Container csi-resizer ready: false, restart count 0 Jan 28 17:51:38.283: INFO: hostexec-i-0d3b431203a4a372c-k7mpl started at 2023-01-28 17:51:33 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:38.284: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:51:38.284: INFO: hostexec-i-0d3b431203a4a372c-5n26x started at 2023-01-28 17:50:27 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:38.284: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:51:38.284: INFO: ebs-csi-node-j2pwn started at 2023-01-28 17:22:13 +0000 UTC (0+3 container statuses recorded) Jan 28 17:51:38.284: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:51:38.284: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:51:38.284: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:51:38.284: INFO: coredns-6dd78b7459-t8lf4 started at 2023-01-28 17:22:48 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:38.284: INFO: Container coredns ready: true, restart count 0 Jan 28 17:51:38.284: INFO: startup-script started at 2023-01-28 17:51:32 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:38.284: INFO: Container startup-script ready: true, restart count 0 Jan 28 17:51:38.284: INFO: pod-2934e8e9-9ba5-4f70-a4c1-3be753776a5f started at 2023-01-28 17:50:33 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:38.284: INFO: Container write-pod ready: true, restart count 0 Jan 28 17:51:38.284: INFO: cilium-cs2hk started at 2023-01-28 17:22:13 +0000 UTC (1+1 container statuses recorded) Jan 28 17:51:38.284: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:51:38.284: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:51:38.284: INFO: netserver-2 started at 2023-01-28 17:50:55 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:38.284: INFO: Container webserver ready: true, restart count 0 Jan 28 17:51:38.284: INFO: pause-pod-596cfff4b4-hd9xp started at 2023-01-28 17:51:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:38.284: INFO: Container agnhost-pause ready: true, restart count 0 Jan 28 17:51:38.284: INFO: csi-mockplugin-0 started at 2023-01-28 17:47:23 +0000 UTC (0+3 container statuses recorded) Jan 28 17:51:38.284: INFO: Container csi-provisioner ready: true, restart count 0 Jan 28 17:51:38.284: INFO: Container driver-registrar ready: true, restart count 0 Jan 28 17:51:38.284: INFO: Container mock ready: true, restart count 0 Jan 28 17:51:38.284: INFO: deployment-8f95b86b8-p9pjw started at 2023-01-28 17:51:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:38.284: INFO: Container nginx ready: false, restart count 0 Jan 28 17:51:39.031: INFO: Latency metrics for node i-0d3b431203a4a372c Jan 28 17:51:39.031: INFO: Logging node info for node i-0e10813507581b097 Jan 28 17:51:39.246: INFO: Node Info: &Node{ObjectMeta:{i-0e10813507581b097 8af9c758-a6f6-4a0d-b634-4d99ed71b929 38608 0 2023-01-28 17:22:07 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e10813507581b097 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0e10813507581b097 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.58 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e10813507581b097"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 17:22:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 17:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 17:22:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 17:40:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0e10813507581b097,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{50478186496 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4078190592 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{45430367772 0} {<nil>} 45430367772 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3973332992 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:34 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:34 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 17:50:34 +0000 UTC,LastTransitionTime:2023-01-28 17:22:07 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 17:50:34 +0000 UTC,LastTransitionTime:2023-01-28 17:22:28 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.58,},NodeAddress{Type:ExternalIP,Address:54.254.187.204,},NodeAddress{Type:InternalDNS,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e10813507581b097.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-187-204.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2922b3ab12fd264b764b33da3e62c2,SystemUUID:ec2922b3-ab12-fd26-4b76-4b33da3e62c2,BootID:07cb73db-252f-45aa-8a98-af3a42c0313b,KernelVersion:4.19.0-23-cloud-amd64,OSImage:Debian GNU/Linux 10 (buster),ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.24.10,KubeProxyVersion:v1.24.10,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.24.10],SizeBytes:112211511,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 k8s.gcr.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 k8s.gcr.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb k8s.gcr.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:818f35653f2e214db81d655063e81995de9073328a3430498624c140881026a3 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1],SizeBytes:21564520,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a k8s.gcr.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 k8s.gcr.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:bb6ed397957e9ca7c65ada0db5c5d1c707c9c8afc80a94acbe69f3ae76988f0c k8s.gcr.io/pause:3.7],SizeBytes:311278,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Jan 28 17:51:39.247: INFO: Logging kubelet events for node i-0e10813507581b097 Jan 28 17:51:39.463: INFO: Logging pods the kubelet thinks is on node i-0e10813507581b097 Jan 28 17:51:39.684: INFO: pod-handle-http-request started at 2023-01-28 17:50:51 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:39.684: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:51:39.684: INFO: deployment-8f95b86b8-85kck started at 2023-01-28 17:51:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:39.684: INFO: Container nginx ready: false, restart count 0 Jan 28 17:51:39.684: INFO: cilium-mbfkc started at 2023-01-28 17:22:08 +0000 UTC (1+1 container statuses recorded) Jan 28 17:51:39.684: INFO: Init container clean-cilium-state ready: true, restart count 0 Jan 28 17:51:39.684: INFO: Container cilium-agent ready: true, restart count 0 Jan 28 17:51:39.684: INFO: boom-server started at 2023-01-28 17:48:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:39.684: INFO: Container boom-server ready: true, restart count 0 Jan 28 17:51:39.684: INFO: pause-pod-596cfff4b4-b25wr started at 2023-01-28 17:51:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:39.684: INFO: Container agnhost-pause ready: true, restart count 0 Jan 28 17:51:39.684: INFO: ebs-csi-node-k6rdn started at 2023-01-28 17:22:08 +0000 UTC (0+3 container statuses recorded) Jan 28 17:51:39.684: INFO: Container ebs-plugin ready: true, restart count 0 Jan 28 17:51:39.684: INFO: Container liveness-probe ready: true, restart count 0 Jan 28 17:51:39.684: INFO: Container node-driver-registrar ready: true, restart count 0 Jan 28 17:51:39.684: INFO: downwardapi-volume-08f58add-29be-47fd-bd05-04ae9b5a1d4a started at 2023-01-28 17:51:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:39.684: INFO: Container client-container ready: false, restart count 0 Jan 28 17:51:39.684: INFO: execpod-affinitymrh45 started at 2023-01-28 17:51:35 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:39.684: INFO: Container agnhost-container ready: false, restart count 0 Jan 28 17:51:39.684: INFO: affinity-nodeport-2mdr9 started at 2023-01-28 17:47:06 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:39.684: INFO: Container affinity-nodeport ready: true, restart count 0 Jan 28 17:51:39.684: INFO: netserver-3 started at 2023-01-28 17:50:55 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:39.684: INFO: Container webserver ready: false, restart count 0 Jan 28 17:51:39.684: INFO: deployment-8f95b86b8-nx48d started at 2023-01-28 17:51:34 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:39.684: INFO: Container nginx ready: false, restart count 0 Jan 28 17:51:39.684: INFO: coredns-6dd78b7459-qk7rr started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:39.684: INFO: Container coredns ready: true, restart count 0 Jan 28 17:51:39.684: INFO: hostexec-i-0e10813507581b097-plljv started at 2023-01-28 17:51:35 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:39.684: INFO: Container agnhost-container ready: true, restart count 0 Jan 28 17:51:39.684: INFO: coredns-autoscaler-5f76bf6c7b-nnmkq started at 2023-01-28 17:22:28 +0000 UTC (0+1 container statuses recorded) Jan 28 17:51:39.684: INFO: Container autoscaler ready: true, restart count 0 Jan 28 17:51:40.495: INFO: Latency metrics for node i-0e10813507581b097 Jan 28 17:51:40.495: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "mount-propagation-9877" for this suite.
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[sig\-node\]\sPods\sshould\sbe\supdated\s\[NodeConformance\]\s\[Conformance\]$'
test/e2e/framework/framework.go:652 Jan 28 17:44:38.569: Unexpected error: <*errors.errorString | 0xc00021e1e0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred test/e2e/framework/pods.go:107
[BeforeEach] [sig-node] Pods test/e2e/framework/framework.go:187 �[1mSTEP�[0m: Creating a kubernetes client Jan 28 17:39:36.181: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename pods �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace �[1mSTEP�[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [BeforeEach] [sig-node] Pods test/e2e/common/node/pods.go:191 [It] should be updated [NodeConformance] [Conformance] test/e2e/framework/framework.go:652 �[1mSTEP�[0m: creating the pod �[1mSTEP�[0m: submitting the pod to kubernetes Jan 28 17:39:38.135: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:40.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:42.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:44.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:46.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:48.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:50.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:52.354: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:54.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:56.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:39:58.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:00.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:02.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:04.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:06.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:08.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:10.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:12.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:14.351: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:16.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:18.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:20.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:22.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:24.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:26.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:28.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:30.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:32.355: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:34.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:36.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:38.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:40.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:42.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:44.354: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:46.360: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:48.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:50.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:52.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:54.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:56.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:40:58.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:00.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:02.354: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:04.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:06.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:08.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:10.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:12.354: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:14.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:16.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:18.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:20.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:22.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:24.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:26.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:28.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:30.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:32.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:34.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:36.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:38.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:40.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:42.354: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:44.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:46.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:48.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:50.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:52.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:54.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:56.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:41:58.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:00.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:02.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:04.353: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:06.352: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:08.354: INFO: The status of Pod pod-update-224dfa31-762a-41f6-8811-10c6c77b136a is Pending, waiting for it to be Running (with Ready = true) Jan 28 17:42:10.352: INFO: The status of Pod pod-update