kubernetes/kops PR #14619 pull-kops-e2e-cni-cilium-eni #1595444626909564928
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | glomikuz: [cilium - eni IPAM MODE] problem with connection from POD to external resource in additional subnet after kops upgrade from 1.23 -> 1.25 #14575 |
| Result | FAILURE |
| Tests | 51 failed / 820 succeeded |
| Started | |
| Elapsed | 56m58s |
| Revision | 8736e0d3342c5f953e9e18830daaf373fad3c231 |
| Refs |
14619 |
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(block\svolmode\)\(allowExpansion\)\]\svolume\-expand\sVerify\sif\soffline\sPVC\sexpansion\sworks$'
test/e2e/storage/testsuites/volume_expand.go:198 k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func4() test/e2e/storage/testsuites/volume_expand.go:198 +0x3bafrom junit_01.xml
{"msg":"FAILED External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand Verify if offline PVC expansion works","completed":39,"skipped":354,"failed":2,"failures":["External Storage [Driver: ebs.csi.aws.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should support expansion of pvcs created for ephemeral pvcs","External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand Verify if offline PVC expansion works"]} [BeforeEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand test/e2e/framework/framework.go:186 �[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m11/23/22 16:33:30.763�[0m Nov 23 16:33:30.763: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP:�[0m Building a namespace api object, basename volume-expand �[38;5;243m11/23/22 16:33:30.764�[0m �[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m11/23/22 16:33:30.954�[0m �[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m11/23/22 16:33:31.077�[0m [It] Verify if offline PVC expansion works test/e2e/storage/testsuites/volume_expand.go:176 Nov 23 16:33:31.201: INFO: Creating resource for dynamic PV Nov 23 16:33:31.201: INFO: Using claimSize:1Gi, test suite supported size:{ 1Gi}, driver(ebs.csi.aws.com) supported size:{ 1Gi} �[1mSTEP:�[0m creating a StorageClass volume-expand-8951-e2e-scgnvn7 �[38;5;243m11/23/22 16:33:31.201�[0m �[1mSTEP:�[0m creating a claim �[38;5;243m11/23/22 16:33:31.265�[0m �[1mSTEP:�[0m Creating a pod with dynamically provisioned volume �[38;5;243m11/23/22 16:33:31.4�[0m Nov 23 16:33:31.473: INFO: Waiting up to 5m0s for pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842" in namespace "volume-expand-8951" to be "running" Nov 23 16:33:31.544: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 70.428105ms Nov 23 16:33:33.612: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2.138582333s Nov 23 16:33:35.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4.134092552s Nov 23 16:33:37.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 6.134426914s Nov 23 16:33:39.614: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 8.140757775s Nov 23 16:33:41.610: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 10.136200669s Nov 23 16:33:43.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 12.134056529s Nov 23 16:33:45.612: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 14.138487665s Nov 23 16:33:47.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 16.134422628s Nov 23 16:33:49.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 18.134332239s Nov 23 16:33:51.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 20.134255991s Nov 23 16:33:53.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 22.133937272s Nov 23 16:33:55.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 24.133961012s Nov 23 16:33:57.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 26.13395894s Nov 23 16:33:59.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 28.134303971s Nov 23 16:34:01.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 30.133764629s Nov 23 16:34:03.611: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 32.137196628s Nov 23 16:34:05.611: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 34.13765498s Nov 23 16:34:07.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 36.133785498s Nov 23 16:34:09.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 38.13393992s Nov 23 16:34:11.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 40.133796239s Nov 23 16:34:13.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 42.135817822s Nov 23 16:34:15.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 44.134324233s Nov 23 16:34:17.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 46.134184125s Nov 23 16:34:19.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 48.133883476s Nov 23 16:34:21.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 50.134458587s Nov 23 16:34:23.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 52.133245987s Nov 23 16:34:25.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 54.134172809s Nov 23 16:34:27.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 56.134326266s Nov 23 16:34:29.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 58.135558611s Nov 23 16:34:31.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.133424187s Nov 23 16:34:33.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.134501094s Nov 23 16:34:35.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.134251737s Nov 23 16:34:37.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.134413307s Nov 23 16:34:39.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.134400279s Nov 23 16:34:41.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.13427034s Nov 23 16:34:43.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.133553182s Nov 23 16:34:45.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.133775721s Nov 23 16:34:47.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.13401327s Nov 23 16:34:49.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.134441716s Nov 23 16:34:51.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.134669656s Nov 23 16:34:53.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.134544131s Nov 23 16:34:55.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.134023731s Nov 23 16:34:57.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.134390643s Nov 23 16:34:59.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.134525707s Nov 23 16:35:01.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.134448195s Nov 23 16:35:03.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.133703446s Nov 23 16:35:05.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.134059083s Nov 23 16:35:07.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.134503893s Nov 23 16:35:09.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m38.134624319s Nov 23 16:35:11.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m40.134034052s Nov 23 16:35:13.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m42.133909519s Nov 23 16:35:15.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.134313469s Nov 23 16:35:17.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.134575715s Nov 23 16:35:19.610: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.136902091s Nov 23 16:35:21.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m50.134539981s Nov 23 16:35:23.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m52.134496683s Nov 23 16:35:25.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m54.134654048s Nov 23 16:35:27.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m56.134534712s Nov 23 16:35:29.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 1m58.134714893s Nov 23 16:35:31.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m0.134224498s Nov 23 16:35:33.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m2.134678309s Nov 23 16:35:35.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m4.134603381s Nov 23 16:35:37.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.134136088s Nov 23 16:35:39.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.134479259s Nov 23 16:35:41.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.133847177s Nov 23 16:35:43.607: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.133712987s Nov 23 16:35:45.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.135584396s Nov 23 16:35:47.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.135387377s Nov 23 16:35:49.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.13505349s Nov 23 16:35:51.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.135326261s Nov 23 16:35:53.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.135483255s Nov 23 16:35:55.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.13445196s Nov 23 16:35:57.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.135379627s Nov 23 16:35:59.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.13562254s Nov 23 16:36:01.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.135165004s Nov 23 16:36:03.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.134383034s Nov 23 16:36:05.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.135537585s Nov 23 16:36:07.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.135243546s Nov 23 16:36:09.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.135310224s Nov 23 16:36:11.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.135580152s Nov 23 16:36:13.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.135309597s Nov 23 16:36:15.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.135097016s Nov 23 16:36:17.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.135710224s Nov 23 16:36:19.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.135088695s Nov 23 16:36:21.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.13534794s Nov 23 16:36:23.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.135432681s Nov 23 16:36:25.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.135136931s Nov 23 16:36:27.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.135561621s Nov 23 16:36:29.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 2m58.135594227s Nov 23 16:36:31.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m0.134630805s Nov 23 16:36:33.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m2.135167195s Nov 23 16:36:35.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.135457024s Nov 23 16:36:37.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m6.135661s Nov 23 16:36:39.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m8.135332259s Nov 23 16:36:41.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.135549693s Nov 23 16:36:43.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.134072346s Nov 23 16:36:45.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m14.13554985s Nov 23 16:36:47.610: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m16.136273848s Nov 23 16:36:49.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m18.135654681s Nov 23 16:36:51.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m20.135137551s Nov 23 16:36:53.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m22.135237949s Nov 23 16:36:55.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.135328412s Nov 23 16:36:57.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.135395374s Nov 23 16:36:59.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.135667019s Nov 23 16:37:01.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.135089411s Nov 23 16:37:03.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.135430535s Nov 23 16:37:05.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m34.135853198s Nov 23 16:37:07.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m36.135129275s Nov 23 16:37:09.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m38.135517012s Nov 23 16:37:11.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m40.135626602s Nov 23 16:37:13.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m42.135085455s Nov 23 16:37:15.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m44.135125105s Nov 23 16:37:17.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m46.135351376s Nov 23 16:37:19.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m48.135081349s Nov 23 16:37:21.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m50.135466008s Nov 23 16:37:23.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m52.134747146s Nov 23 16:37:25.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m54.135188892s Nov 23 16:37:27.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m56.135623195s Nov 23 16:37:29.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 3m58.134953774s Nov 23 16:37:31.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m0.135152974s Nov 23 16:37:33.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m2.135226241s Nov 23 16:37:35.610: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m4.136063894s Nov 23 16:37:37.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.134583236s Nov 23 16:37:39.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.135397024s Nov 23 16:37:41.611: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.137825058s Nov 23 16:37:43.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.135398493s Nov 23 16:37:45.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.134479678s Nov 23 16:37:47.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m16.135431745s Nov 23 16:37:49.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m18.135320624s Nov 23 16:37:51.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m20.135372016s Nov 23 16:37:53.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m22.134536071s Nov 23 16:37:55.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m24.134618644s Nov 23 16:37:57.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.13536868s Nov 23 16:37:59.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.135579536s Nov 23 16:38:01.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.134700596s Nov 23 16:38:03.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.135596273s Nov 23 16:38:05.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.135176569s Nov 23 16:38:07.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.13470513s Nov 23 16:38:09.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.135097154s Nov 23 16:38:11.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.135294708s Nov 23 16:38:13.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.134456742s Nov 23 16:38:15.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.135489579s Nov 23 16:38:17.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.13497987s Nov 23 16:38:19.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.135301671s Nov 23 16:38:21.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.135433358s Nov 23 16:38:23.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.134868106s Nov 23 16:38:25.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.13468022s Nov 23 16:38:27.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.135289366s Nov 23 16:38:29.609: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.135590326s Nov 23 16:38:31.608: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.134631346s Nov 23 16:38:31.672: INFO: Pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.198736453s Nov 23 16:38:31.672: INFO: Unexpected error: While creating pods for resizing: <*errors.errorString | 0xc002eb0f90>: { s: "pod \"pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842\" is not Running: timed out while waiting for pod volume-expand-8951/pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842 to be running", } Nov 23 16:38:31.672: FAIL: While creating pods for resizing: pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842" is not Running: timed out while waiting for pod volume-expand-8951/pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842 to be running Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func4() test/e2e/storage/testsuites/volume_expand.go:198 +0x3ba Nov 23 16:38:31.673: INFO: Deleting pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842" in namespace "volume-expand-8951" Nov 23 16:38:31.741: INFO: Wait up to 5m0s for pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842" to be fully deleted �[1mSTEP:�[0m Deleting pod �[38;5;243m11/23/22 16:38:31.805�[0m Nov 23 16:38:31.805: INFO: Deleting pod "pod-a6e1a6d4-5ce4-4a2a-898e-48654544c842" in namespace "volume-expand-8951" �[1mSTEP:�[0m Deleting pvc �[38;5;243m11/23/22 16:38:31.869�[0m Nov 23 16:38:31.933: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comnnrll" �[1mSTEP:�[0m Deleting sc �[38;5;243m11/23/22 16:38:31.998�[0m [AfterEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand test/e2e/framework/framework.go:187 �[1mSTEP:�[0m Collecting events from namespace "volume-expand-8951". �[38;5;243m11/23/22 16:38:32.063�[0m �[1mSTEP:�[0m Found 4 events. �[38;5;243m11/23/22 16:38:32.126�[0m Nov 23 16:38:32.127: INFO: At 2022-11-23 16:33:31 +0000 UTC - event for ebs.csi.aws.comnnrll: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 16:38:32.127: INFO: At 2022-11-23 16:33:31 +0000 UTC - event for ebs.csi.aws.comnnrll: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-8951/ebs.csi.aws.comnnrll" Nov 23 16:38:32.127: INFO: At 2022-11-23 16:33:31 +0000 UTC - event for ebs.csi.aws.comnnrll: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 16:38:32.127: INFO: At 2022-11-23 16:33:41 +0000 UTC - event for ebs.csi.aws.comnnrll: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-8951-e2e-scgnvn7": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 16:38:32.190: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 16:38:32.190: INFO: Nov 23 16:38:32.255: INFO: Logging node info for node i-060d86c6ffc5622b0 Nov 23 16:38:32.319: INFO: Node Info: &Node{ObjectMeta:{i-060d86c6ffc5622b0 b49ff0b3-8686-4c0f-b991-1d427ca43e2a 49326 0 2022-11-23 16:02:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-060d86c6ffc5622b0 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.40.57 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-060d86c6ffc5622b0"} io.cilium.network.ipv4-cilium-host:172.20.57.204 io.cilium.network.ipv4-health-ip:172.20.37.200 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 16:02:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2022-11-23 16:03:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:03:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:35:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-060d86c6ffc5622b0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:04:42 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:35:44 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:35:44 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:35:44 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:35:44 +0000 UTC,LastTransitionTime:2022-11-23 16:03:24 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.57,},NodeAddress{Type:ExternalIP,Address:54.200.161.23,},NodeAddress{Type:InternalDNS,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-200-161-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec226e0d77c347467223a292b33eb0c5,SystemUUID:ec226e0d-77c3-4746-7223-a292b33eb0c5,BootID:936b9dd2-17e7-49ac-8c1e-7556bf2c9ee8,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.25.4],SizeBytes:129085904,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.25.4],SizeBytes:118436153,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.25.4],SizeBytes:51921208,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42928334,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41477333,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:85d3f1e9dacc72531445989bb10999e1e70ebc409d11be57e5baa5f031a893b0 registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1],SizeBytes:18257577,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955428,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:38:32.319: INFO: Logging kubelet events for node i-060d86c6ffc5622b0 Nov 23 16:38:32.386: INFO: Logging pods the kubelet thinks is on node i-060d86c6ffc5622b0 Nov 23 16:38:32.521: INFO: etcd-manager-events-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:38:32.521: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 16:38:32.521: INFO: kops-controller-7w98n started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:38:32.521: INFO: Container kops-controller ready: true, restart count 0 Nov 23 16:38:32.521: INFO: kube-scheduler-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:38:32.521: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 16:38:32.521: INFO: dns-controller-89c6d5f67-9hm78 started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:38:32.521: INFO: Container dns-controller ready: true, restart count 0 Nov 23 16:38:32.521: INFO: cilium-f2gg7 started at 2022-11-23 16:03:06 +0000 UTC (1+1 container statuses recorded) Nov 23 16:38:32.521: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:38:32.521: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 16:38:32.521: INFO: cilium-operator-7f7bcdf787-xnz2q started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:38:32.521: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 16:38:32.521: INFO: aws-cloud-controller-manager-dzzkp started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:38:32.521: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Nov 23 16:38:32.521: INFO: etcd-manager-main-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:38:32.521: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 16:38:32.521: INFO: kube-apiserver-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+2 container statuses recorded) Nov 23 16:38:32.521: INFO: Container healthcheck ready: true, restart count 0 Nov 23 16:38:32.521: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 16:38:32.521: INFO: kube-controller-manager-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:38:32.521: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 16:38:32.521: INFO: ebs-csi-node-vpv4t started at 2022-11-23 16:03:06 +0000 UTC (0+3 container statuses recorded) Nov 23 16:38:32.521: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:38:32.521: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:38:32.521: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:38:32.768: INFO: Latency metrics for node i-060d86c6ffc5622b0 Nov 23 16:38:32.768: INFO: Logging node info for node i-0765e732a9635923b Nov 23 16:38:32.832: INFO: Node Info: &Node{ObjectMeta:{i-0765e732a9635923b 10981661-cc27-412a-9e66-7a37930b898f 48852 0 2022-11-23 16:04:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0765e732a9635923b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0765e732a9635923b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.58.74 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0765e732a9635923b"} io.cilium.network.ipv4-cilium-host:172.20.54.198 io.cilium.network.ipv4-health-ip:172.20.60.24 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:07 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:34:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0765e732a9635923b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:07 +0000 UTC,LastTransitionTime:2022-11-23 16:05:07 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:34:04 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:34:04 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:34:04 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:34:04 +0000 UTC,LastTransitionTime:2022-11-23 16:05:00 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.58.74,},NodeAddress{Type:ExternalIP,Address:52.25.125.23,},NodeAddress{Type:InternalDNS,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-25-125-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec23d129b4036d8d979053444dfae008,SystemUUID:ec23d129-b403-6d8d-9790-53444dfae008,BootID:fe390540-4601-456e-9686-93a36ea3dc8f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:38:32.832: INFO: Logging kubelet events for node i-0765e732a9635923b Nov 23 16:38:32.899: INFO: Logging pods the kubelet thinks is on node i-0765e732a9635923b Nov 23 16:38:32.971: INFO: coredns-6c7bddbb75-2n2dw started at 2022-11-23 16:05:19 +0000 UTC (0+1 container statuses recorded) Nov 23 16:38:32.971: INFO: Container coredns ready: true, restart count 0 Nov 23 16:38:32.971: INFO: ebs-csi-node-5khlc started at 2022-11-23 16:04:40 +0000 UTC (0+3 container statuses recorded) Nov 23 16:38:32.971: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:38:32.971: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:38:32.971: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:38:32.971: INFO: cilium-czwdq started at 2022-11-23 16:04:40 +0000 UTC (1+1 container statuses recorded) Nov 23 16:38:32.971: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:38:32.971: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:38:33.294: INFO: Latency metrics for node i-0765e732a9635923b Nov 23 16:38:33.295: INFO: Logging node info for node i-0ade3c3082581cb52 Nov 23 16:38:33.358: INFO: Node Info: &Node{ObjectMeta:{i-0ade3c3082581cb52 2305662a-7d05-4979-bcec-220b8b4672db 48410 0 2022-11-23 16:04:42 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0ade3c3082581cb52 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0ade3c3082581cb52 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.56.12 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0ade3c3082581cb52"} io.cilium.network.ipv4-cilium-host:172.20.45.237 io.cilium.network.ipv4-health-ip:172.20.55.90 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:33:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0ade3c3082581cb52,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:12 +0000 UTC,LastTransitionTime:2022-11-23 16:05:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:33:47 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:33:47 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:33:47 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:33:47 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.56.12,},NodeAddress{Type:ExternalIP,Address:35.92.243.169,},NodeAddress{Type:InternalDNS,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-92-243-169.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bd980dfb52643cb4857d57b45824b,SystemUUID:ec2bd980-dfb5-2643-cb48-57d57b45824b,BootID:55912341-dce8-4199-a57c-c8703e24af46,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c registry.k8s.io/e2e-test-images/nautilus:1.5],SizeBytes:49642095,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 registry.k8s.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:38:33.359: INFO: Logging kubelet events for node i-0ade3c3082581cb52 Nov 23 16:38:33.427: INFO: Logging pods the kubelet thinks is on node i-0ade3c3082581cb52 Nov 23 16:38:33.500: INFO: cilium-q5tsk started at 2022-11-23 16:04:43 +0000 UTC (1+1 container statuses recorded) Nov 23 16:38:33.500: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:38:33.500: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:38:33.500: INFO: ebs-csi-node-t56sd started at 2022-11-23 16:04:43 +0000 UTC (0+3 container statuses recorded) Nov 23 16:38:33.500: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:38:33.500: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:38:33.500: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:38:33.793: INFO: Latency metrics for node i-0ade3c3082581cb52 Nov 23 16:38:33.793: INFO: Logging node info for node i-0bf2756d84bae9168 Nov 23 16:38:33.857: INFO: Node Info: &Node{ObjectMeta:{i-0bf2756d84bae9168 2f26f127-5eb0-44f3-97f0-990045e7b900 48892 0 2022-11-23 16:04:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0bf2756d84bae9168 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0bf2756d84bae9168 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.218 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0bf2756d84bae9168"} io.cilium.network.ipv4-cilium-host:172.20.35.201 io.cilium.network.ipv4-health-ip:172.20.51.170 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:02 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:34:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0bf2756d84bae9168,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:02 +0000 UTC,LastTransitionTime:2022-11-23 16:05:02 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:34:08 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:34:08 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:34:08 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:34:08 +0000 UTC,LastTransitionTime:2022-11-23 16:04:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.218,},NodeAddress{Type:ExternalIP,Address:34.221.37.130,},NodeAddress{Type:InternalDNS,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-37-130.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d7db7f0ff759b7a6209e832f1dcea,SystemUUID:ec2d7db7-f0ff-759b-7a62-09e832f1dcea,BootID:8f294a85-a1ed-4dfb-bdb9-f8818e1f067f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c registry.k8s.io/e2e-test-images/nautilus:1.5],SizeBytes:49642095,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f registry.k8s.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:38:33.857: INFO: Logging kubelet events for node i-0bf2756d84bae9168 Nov 23 16:38:33.924: INFO: Logging pods the kubelet thinks is on node i-0bf2756d84bae9168 Nov 23 16:38:33.996: INFO: ebs-csi-node-crshz started at 2022-11-23 16:04:35 +0000 UTC (0+3 container statuses recorded) Nov 23 16:38:33.996: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:38:33.996: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:38:33.996: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:38:33.996: INFO: ebs-csi-controller-847c4b8bc4-kwpsb started at 2022-11-23 16:04:56 +0000 UTC (0+5 container statuses recorded) Nov 23 16:38:33.996: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:38:33.996: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:38:33.996: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:38:33.996: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:38:33.996: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:38:33.996: INFO: cilium-scm24 started at 2022-11-23 16:04:35 +0000 UTC (1+1 container statuses recorded) Nov 23 16:38:33.996: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:38:33.996: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:38:34.308: INFO: Latency metrics for node i-0bf2756d84bae9168 Nov 23 16:38:34.308: INFO: Logging node info for node i-0fe7a5031fb82cc5b Nov 23 16:38:34.372: INFO: Node Info: &Node{ObjectMeta:{i-0fe7a5031fb82cc5b 0297efe3-5070-45f1-8aa2-9a1056b17b59 47700 0 2022-11-23 16:04:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0fe7a5031fb82cc5b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0fe7a5031fb82cc5b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.89 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0fe7a5031fb82cc5b"} io.cilium.network.ipv4-cilium-host:172.20.57.57 io.cilium.network.ipv4-health-ip:172.20.59.99 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-11-23 16:04:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:03 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:33:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0fe7a5031fb82cc5b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:03 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:33:40 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:33:40 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:33:40 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:33:40 +0000 UTC,LastTransitionTime:2022-11-23 16:04:54 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.89,},NodeAddress{Type:ExternalIP,Address:34.221.99.167,},NodeAddress{Type:InternalDNS,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-99-167.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29292cb2c053c780af3b39ea2fa8c6,SystemUUID:ec29292c-b2c0-53c7-80af-3b39ea2fa8c6,BootID:ff4df73e-ccb2-4109-99ca-04a1b7ab5611,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c registry.k8s.io/e2e-test-images/nautilus:1.5],SizeBytes:49642095,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 registry.k8s.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:38:34.372: INFO: Logging kubelet events for node i-0fe7a5031fb82cc5b Nov 23 16:38:34.439: INFO: Logging pods the kubelet thinks is on node i-0fe7a5031fb82cc5b Nov 23 16:38:34.517: INFO: ebs-csi-node-8ffwl started at 2022-11-23 16:04:34 +0000 UTC (0+3 container statuses recorded) Nov 23 16:38:34.517: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:38:34.517: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:38:34.517: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:38:34.517: INFO: cilium-727pv started at 2022-11-23 16:04:34 +0000 UTC (1+1 container statuses recorded) Nov 23 16:38:34.517: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:38:34.517: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:38:34.517: INFO: coredns-6c7bddbb75-bpdc6 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded) Nov 23 16:38:34.517: INFO: Container coredns ready: true, restart count 0 Nov 23 16:38:34.517: INFO: coredns-autoscaler-db7b97744-r4jm7 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded) Nov 23 16:38:34.517: INFO: Container autoscaler ready: true, restart count 0 Nov 23 16:38:34.517: INFO: ebs-csi-controller-847c4b8bc4-lxdkr started at 2022-11-23 16:04:54 +0000 UTC (0+5 container statuses recorded) Nov 23 16:38:34.517: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:38:34.517: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:38:34.517: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:38:34.517: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:38:34.517: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:38:34.828: INFO: Latency metrics for node i-0fe7a5031fb82cc5b Nov 23 16:38:34.828: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP:�[0m Destroying namespace "volume-expand-8951" for this suite. �[38;5;243m11/23/22 16:38:34.893�[0m
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(block\svolmode\)\(allowExpansion\)\]\svolume\-expand\sshould\sresize\svolume\swhen\sPVC\sis\sedited\swhile\spod\sis\susing\sit$'
test/e2e/storage/testsuites/volume_expand.go:274 k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func5() test/e2e/storage/testsuites/volume_expand.go:274 +0x3b3from junit_01.xml
{"msg":"FAILED External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand should resize volume when PVC is edited while pod is using it","completed":4,"skipped":16,"failed":1,"failures":["External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand should resize volume when PVC is edited while pod is using it"]} [BeforeEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand test/e2e/framework/framework.go:186 �[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m11/23/22 16:10:27.394�[0m Nov 23 16:10:27.394: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP:�[0m Building a namespace api object, basename volume-expand �[38;5;243m11/23/22 16:10:27.395�[0m �[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m11/23/22 16:10:27.583�[0m �[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m11/23/22 16:10:27.707�[0m [It] should resize volume when PVC is edited while pod is using it test/e2e/storage/testsuites/volume_expand.go:252 Nov 23 16:10:27.831: INFO: Creating resource for dynamic PV Nov 23 16:10:27.831: INFO: Using claimSize:1Gi, test suite supported size:{ 1Gi}, driver(ebs.csi.aws.com) supported size:{ 1Gi} �[1mSTEP:�[0m creating a StorageClass volume-expand-465-e2e-scpr24g �[38;5;243m11/23/22 16:10:27.832�[0m �[1mSTEP:�[0m creating a claim �[38;5;243m11/23/22 16:10:27.895�[0m �[1mSTEP:�[0m Creating a pod with dynamically provisioned volume �[38;5;243m11/23/22 16:10:28.024�[0m Nov 23 16:10:28.088: INFO: Waiting up to 5m0s for pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c" in namespace "volume-expand-465" to be "running" Nov 23 16:10:28.152: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 64.435594ms Nov 23 16:10:30.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2.128066139s Nov 23 16:10:32.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4.127826049s Nov 23 16:10:34.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 6.127660367s Nov 23 16:10:36.217: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 8.128892515s Nov 23 16:10:38.217: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 10.128841597s Nov 23 16:10:40.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 12.12725498s Nov 23 16:10:42.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 14.127691897s Nov 23 16:10:44.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 16.128325144s Nov 23 16:10:46.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 18.127798824s Nov 23 16:10:48.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 20.127559583s Nov 23 16:10:50.236: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 22.147842046s Nov 23 16:10:52.220: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 24.131910424s Nov 23 16:10:54.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 26.127837298s Nov 23 16:10:56.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 28.127363712s Nov 23 16:10:58.221: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 30.133626029s Nov 23 16:11:00.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 32.127738434s Nov 23 16:11:02.217: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 34.129104902s Nov 23 16:11:04.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 36.128368865s Nov 23 16:11:06.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 38.128585168s Nov 23 16:11:08.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 40.127713999s Nov 23 16:11:10.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 42.128471748s Nov 23 16:11:12.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 44.127838949s Nov 23 16:11:14.217: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 46.129284023s Nov 23 16:11:16.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 48.127645996s Nov 23 16:11:18.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 50.127410075s Nov 23 16:11:20.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 52.128661708s Nov 23 16:11:22.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 54.1277104s Nov 23 16:11:24.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 56.127402073s Nov 23 16:11:26.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 58.127508258s Nov 23 16:11:28.220: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.132179697s Nov 23 16:11:30.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.127276623s Nov 23 16:11:32.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.127593971s Nov 23 16:11:34.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.128270651s Nov 23 16:11:36.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.127772042s Nov 23 16:11:38.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.127736384s Nov 23 16:11:40.218: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.130460098s Nov 23 16:11:42.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.128492274s Nov 23 16:11:44.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.127971856s Nov 23 16:11:46.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.127560026s Nov 23 16:11:48.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.127865866s Nov 23 16:11:50.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.128359421s Nov 23 16:11:52.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.127966389s Nov 23 16:11:54.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.128005813s Nov 23 16:11:56.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.128024686s Nov 23 16:11:58.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.1284649s Nov 23 16:12:00.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.127804888s Nov 23 16:12:02.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.128602187s Nov 23 16:12:04.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.128283612s Nov 23 16:12:06.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m38.128584099s Nov 23 16:12:08.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m40.127940148s Nov 23 16:12:10.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m42.128183277s Nov 23 16:12:12.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.12841992s Nov 23 16:12:14.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.127503654s Nov 23 16:12:16.218: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.130092829s Nov 23 16:12:18.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m50.127426906s Nov 23 16:12:20.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m52.127359873s Nov 23 16:12:22.218: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m54.129942404s Nov 23 16:12:24.217: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m56.129677877s Nov 23 16:12:26.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 1m58.128600856s Nov 23 16:12:28.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m0.127689935s Nov 23 16:12:30.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m2.12792284s Nov 23 16:12:32.218: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m4.129813847s Nov 23 16:12:34.219: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.130796591s Nov 23 16:12:36.218: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.130278878s Nov 23 16:12:38.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.12669315s Nov 23 16:12:40.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.127185756s Nov 23 16:12:42.219: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.131121444s Nov 23 16:12:44.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.126479329s Nov 23 16:12:46.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.127272157s Nov 23 16:12:48.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.126926613s Nov 23 16:12:50.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.126549068s Nov 23 16:12:52.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.127052343s Nov 23 16:12:54.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.126491271s Nov 23 16:12:56.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.12832642s Nov 23 16:12:58.217: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.128802016s Nov 23 16:13:00.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.127171736s Nov 23 16:13:02.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.126504283s Nov 23 16:13:04.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.126080977s Nov 23 16:13:06.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.12602066s Nov 23 16:13:08.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.12664947s Nov 23 16:13:10.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.126571678s Nov 23 16:13:12.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.126447852s Nov 23 16:13:14.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.12611742s Nov 23 16:13:16.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.127312315s Nov 23 16:13:18.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.126423162s Nov 23 16:13:20.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.128599858s Nov 23 16:13:22.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.127933232s Nov 23 16:13:24.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.127087162s Nov 23 16:13:26.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 2m58.126924786s Nov 23 16:13:28.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m0.126842366s Nov 23 16:13:30.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m2.126889123s Nov 23 16:13:32.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.127371114s Nov 23 16:13:34.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m6.12795565s Nov 23 16:13:36.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m8.127016552s Nov 23 16:13:38.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.126429281s Nov 23 16:13:40.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.127060137s Nov 23 16:13:42.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m14.126703117s Nov 23 16:13:44.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m16.127303585s Nov 23 16:13:46.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m18.12718569s Nov 23 16:13:48.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m20.127260139s Nov 23 16:13:50.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m22.127386356s Nov 23 16:13:52.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.127355039s Nov 23 16:13:54.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.126118614s Nov 23 16:13:56.218: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.129783663s Nov 23 16:13:58.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.126941465s Nov 23 16:14:00.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.126923523s Nov 23 16:14:02.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m34.127290534s Nov 23 16:14:04.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m36.126447832s Nov 23 16:14:06.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m38.126116664s Nov 23 16:14:08.217: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m40.12968485s Nov 23 16:14:10.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m42.126262087s Nov 23 16:14:12.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m44.127480855s Nov 23 16:14:14.225: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m46.136892996s Nov 23 16:14:16.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m48.127445062s Nov 23 16:14:18.218: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m50.129869312s Nov 23 16:14:20.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m52.127622971s Nov 23 16:14:22.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m54.127202486s Nov 23 16:14:24.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m56.126282794s Nov 23 16:14:26.218: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 3m58.130693951s Nov 23 16:14:28.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m0.126564132s Nov 23 16:14:30.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m2.126968852s Nov 23 16:14:32.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m4.126765713s Nov 23 16:14:34.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.126599302s Nov 23 16:14:36.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.126473756s Nov 23 16:14:38.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.126699021s Nov 23 16:14:40.217: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.12914246s Nov 23 16:14:42.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.126810555s Nov 23 16:14:44.219: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m16.130902687s Nov 23 16:14:46.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m18.128029698s Nov 23 16:14:48.216: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m20.127792934s Nov 23 16:14:50.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m22.126472375s Nov 23 16:14:52.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m24.126459106s Nov 23 16:14:54.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.126704036s Nov 23 16:14:56.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.126724075s Nov 23 16:14:58.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.126288051s Nov 23 16:15:00.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.126526639s Nov 23 16:15:02.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.126659182s Nov 23 16:15:04.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.126313093s Nov 23 16:15:06.224: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.136007871s Nov 23 16:15:08.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.125996325s Nov 23 16:15:10.217: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.128916155s Nov 23 16:15:12.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.12756252s Nov 23 16:15:14.248: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.159967814s Nov 23 16:15:16.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.12681583s Nov 23 16:15:18.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.126501219s Nov 23 16:15:20.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.126714362s Nov 23 16:15:22.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.127079652s Nov 23 16:15:24.217: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.12895535s Nov 23 16:15:26.215: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.126946149s Nov 23 16:15:28.214: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.126267267s Nov 23 16:15:28.276: INFO: Pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.187929708s Nov 23 16:15:28.276: INFO: Unexpected error: While creating pods for resizing: <*errors.errorString | 0xc001075c40>: { s: "pod \"pod-43c92ca4-6100-4962-8f06-49c41a40f30c\" is not Running: timed out while waiting for pod volume-expand-465/pod-43c92ca4-6100-4962-8f06-49c41a40f30c to be running", } Nov 23 16:15:28.276: FAIL: While creating pods for resizing: pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c" is not Running: timed out while waiting for pod volume-expand-465/pod-43c92ca4-6100-4962-8f06-49c41a40f30c to be running Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func5() test/e2e/storage/testsuites/volume_expand.go:274 +0x3b3 Nov 23 16:15:28.276: INFO: Deleting pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c" in namespace "volume-expand-465" Nov 23 16:15:28.341: INFO: Wait up to 5m0s for pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c" to be fully deleted �[1mSTEP:�[0m Deleting pod �[38;5;243m11/23/22 16:15:28.402�[0m Nov 23 16:15:28.403: INFO: Deleting pod "pod-43c92ca4-6100-4962-8f06-49c41a40f30c" in namespace "volume-expand-465" �[1mSTEP:�[0m Deleting pvc �[38;5;243m11/23/22 16:15:28.464�[0m Nov 23 16:15:28.526: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comblmdl" �[1mSTEP:�[0m Deleting sc �[38;5;243m11/23/22 16:15:28.589�[0m [AfterEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand test/e2e/framework/framework.go:187 �[1mSTEP:�[0m Collecting events from namespace "volume-expand-465". �[38;5;243m11/23/22 16:15:28.652�[0m �[1mSTEP:�[0m Found 5 events. �[38;5;243m11/23/22 16:15:28.714�[0m Nov 23 16:15:28.714: INFO: At 2022-11-23 16:10:27 +0000 UTC - event for ebs.csi.aws.comblmdl: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 16:15:28.714: INFO: At 2022-11-23 16:10:28 +0000 UTC - event for ebs.csi.aws.comblmdl: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 16:15:28.714: INFO: At 2022-11-23 16:10:28 +0000 UTC - event for ebs.csi.aws.comblmdl: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-465/ebs.csi.aws.comblmdl" Nov 23 16:15:28.714: INFO: At 2022-11-23 16:10:38 +0000 UTC - event for ebs.csi.aws.comblmdl: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-465-e2e-scpr24g": rpc error: code = Internal desc = Could not create volume "pvc-28d8057b-3d22-42d6-ae5b-f276d8f557fa": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Nov 23 16:15:28.714: INFO: At 2022-11-23 16:10:49 +0000 UTC - event for ebs.csi.aws.comblmdl: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-465-e2e-scpr24g": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 16:15:28.775: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 16:15:28.775: INFO: Nov 23 16:15:28.839: INFO: Logging node info for node i-060d86c6ffc5622b0 Nov 23 16:15:28.901: INFO: Node Info: &Node{ObjectMeta:{i-060d86c6ffc5622b0 b49ff0b3-8686-4c0f-b991-1d427ca43e2a 18859 0 2022-11-23 16:02:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-060d86c6ffc5622b0 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.40.57 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-060d86c6ffc5622b0"} io.cilium.network.ipv4-cilium-host:172.20.57.204 io.cilium.network.ipv4-health-ip:172.20.37.200 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 16:02:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2022-11-23 16:03:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:03:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:15:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-060d86c6ffc5622b0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:04:42 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:15:18 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:15:18 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:15:18 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:15:18 +0000 UTC,LastTransitionTime:2022-11-23 16:03:24 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.57,},NodeAddress{Type:ExternalIP,Address:54.200.161.23,},NodeAddress{Type:InternalDNS,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-200-161-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec226e0d77c347467223a292b33eb0c5,SystemUUID:ec226e0d-77c3-4746-7223-a292b33eb0c5,BootID:936b9dd2-17e7-49ac-8c1e-7556bf2c9ee8,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.25.4],SizeBytes:129085904,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.25.4],SizeBytes:118436153,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.25.4],SizeBytes:51921208,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42928334,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41477333,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:85d3f1e9dacc72531445989bb10999e1e70ebc409d11be57e5baa5f031a893b0 registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1],SizeBytes:18257577,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955428,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:15:28.901: INFO: Logging kubelet events for node i-060d86c6ffc5622b0 Nov 23 16:15:28.973: INFO: Logging pods the kubelet thinks is on node i-060d86c6ffc5622b0 Nov 23 16:15:29.045: INFO: etcd-manager-events-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.045: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 16:15:29.045: INFO: kops-controller-7w98n started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.045: INFO: Container kops-controller ready: true, restart count 0 Nov 23 16:15:29.045: INFO: dns-controller-89c6d5f67-9hm78 started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.045: INFO: Container dns-controller ready: true, restart count 0 Nov 23 16:15:29.045: INFO: cilium-f2gg7 started at 2022-11-23 16:03:06 +0000 UTC (1+1 container statuses recorded) Nov 23 16:15:29.045: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:15:29.045: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 16:15:29.045: INFO: cilium-operator-7f7bcdf787-xnz2q started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.045: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 16:15:29.045: INFO: aws-cloud-controller-manager-dzzkp started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.045: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Nov 23 16:15:29.045: INFO: etcd-manager-main-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.045: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 16:15:29.045: INFO: kube-apiserver-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+2 container statuses recorded) Nov 23 16:15:29.045: INFO: Container healthcheck ready: true, restart count 0 Nov 23 16:15:29.045: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 16:15:29.045: INFO: kube-controller-manager-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.045: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 16:15:29.045: INFO: kube-scheduler-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.045: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 16:15:29.045: INFO: ebs-csi-node-vpv4t started at 2022-11-23 16:03:06 +0000 UTC (0+3 container statuses recorded) Nov 23 16:15:29.045: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:15:29.045: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:15:29.045: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:15:29.303: INFO: Latency metrics for node i-060d86c6ffc5622b0 Nov 23 16:15:29.303: INFO: Logging node info for node i-0765e732a9635923b Nov 23 16:15:29.364: INFO: Node Info: &Node{ObjectMeta:{i-0765e732a9635923b 10981661-cc27-412a-9e66-7a37930b898f 16389 0 2022-11-23 16:04:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0765e732a9635923b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0765e732a9635923b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.58.74 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0765e732a9635923b"} io.cilium.network.ipv4-cilium-host:172.20.54.198 io.cilium.network.ipv4-health-ip:172.20.60.24 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:07 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:14:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0765e732a9635923b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:07 +0000 UTC,LastTransitionTime:2022-11-23 16:05:07 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:14:11 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:14:11 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:14:11 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:14:11 +0000 UTC,LastTransitionTime:2022-11-23 16:05:00 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.58.74,},NodeAddress{Type:ExternalIP,Address:52.25.125.23,},NodeAddress{Type:InternalDNS,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-25-125-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec23d129b4036d8d979053444dfae008,SystemUUID:ec23d129-b403-6d8d-9790-53444dfae008,BootID:fe390540-4601-456e-9686-93a36ea3dc8f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:15:29.365: INFO: Logging kubelet events for node i-0765e732a9635923b Nov 23 16:15:29.429: INFO: Logging pods the kubelet thinks is on node i-0765e732a9635923b Nov 23 16:15:29.495: INFO: test-webserver-b4d69206-2aec-45d1-8fb9-c48caaead6fe started at 2022-11-23 16:14:37 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.495: INFO: Container test-webserver ready: true, restart count 0 Nov 23 16:15:29.495: INFO: ebs-csi-node-5khlc started at 2022-11-23 16:04:40 +0000 UTC (0+3 container statuses recorded) Nov 23 16:15:29.495: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:15:29.495: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:15:29.495: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:15:29.495: INFO: hostexec-i-0765e732a9635923b-5x79n started at 2022-11-23 16:15:23 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.495: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 16:15:29.495: INFO: pod-csi-inline-volumes started at 2022-11-23 16:14:16 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.495: INFO: Container pod-csi-inline-volumes ready: false, restart count 0 Nov 23 16:15:29.495: INFO: test-ss-1 started at 2022-11-23 16:14:25 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.495: INFO: Container webserver ready: true, restart count 0 Nov 23 16:15:29.495: INFO: cilium-czwdq started at 2022-11-23 16:04:40 +0000 UTC (1+1 container statuses recorded) Nov 23 16:15:29.495: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:15:29.495: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:15:29.495: INFO: busybox-scheduling-edc5698f-78b6-4bf7-b078-90425a109df1 started at 2022-11-23 16:15:09 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.495: INFO: Container busybox-scheduling-edc5698f-78b6-4bf7-b078-90425a109df1 ready: true, restart count 0 Nov 23 16:15:29.495: INFO: netserver-0 started at 2022-11-23 16:15:14 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.495: INFO: Container webserver ready: false, restart count 0 Nov 23 16:15:29.495: INFO: coredns-6c7bddbb75-2n2dw started at 2022-11-23 16:05:19 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.495: INFO: Container coredns ready: true, restart count 0 Nov 23 16:15:29.495: INFO: busybox-privileged-true-5245c4e8-9bd4-4a02-b399-4a8e375d2a73 started at 2022-11-23 16:15:28 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:29.495: INFO: Container busybox-privileged-true-5245c4e8-9bd4-4a02-b399-4a8e375d2a73 ready: false, restart count 0 Nov 23 16:15:29.803: INFO: Latency metrics for node i-0765e732a9635923b Nov 23 16:15:29.803: INFO: Logging node info for node i-0ade3c3082581cb52 Nov 23 16:15:29.865: INFO: Node Info: &Node{ObjectMeta:{i-0ade3c3082581cb52 2305662a-7d05-4979-bcec-220b8b4672db 19080 0 2022-11-23 16:04:42 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0ade3c3082581cb52 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0ade3c3082581cb52 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.56.12 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-volume-1299":"i-0ade3c3082581cb52","csi-mock-csi-mock-volumes-4721":"i-0ade3c3082581cb52","ebs.csi.aws.com":"i-0ade3c3082581cb52"} io.cilium.network.ipv4-cilium-host:172.20.45.237 io.cilium.network.ipv4-health-ip:172.20.55.90 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:15:25 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 16:15:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0ade3c3082581cb52,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:12 +0000 UTC,LastTransitionTime:2022-11-23 16:05:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:15:25 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:15:25 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:15:25 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:15:25 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.56.12,},NodeAddress{Type:ExternalIP,Address:35.92.243.169,},NodeAddress{Type:InternalDNS,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-92-243-169.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bd980dfb52643cb4857d57b45824b,SystemUUID:ec2bd980-dfb5-2643-cb48-57d57b45824b,BootID:55912341-dce8-4199-a57c-c8703e24af46,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 registry.k8s.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-volume-1299^fe7ee176-6b49-11ed-bf7b-f6b8f8ee5207],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-1299^fe7ee176-6b49-11ed-bf7b-f6b8f8ee5207,DevicePath:,},},Config:nil,},} Nov 23 16:15:29.865: INFO: Logging kubelet events for node i-0ade3c3082581cb52 Nov 23 16:15:29.929: INFO: Logging pods the kubelet thinks is on node i-0ade3c3082581cb52 Nov 23 16:15:30.003: INFO: csi-mockplugin-0 started at 2022-11-23 16:14:51 +0000 UTC (0+3 container statuses recorded) Nov 23 16:15:30.003: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:15:30.003: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 16:15:30.003: INFO: Container mock ready: true, restart count 0 Nov 23 16:15:30.003: INFO: pod-ephm-test-configmap-r7fl started at 2022-11-23 16:13:45 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:30.003: INFO: Container test-container-subpath-configmap-r7fl ready: false, restart count 0 Nov 23 16:15:30.003: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 16:14:51 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:30.003: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:15:30.003: INFO: csi-mockplugin-resizer-0 started at 2022-11-23 16:14:51 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:30.003: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:15:30.003: INFO: test-ss-0 started at 2022-11-23 16:13:25 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:30.003: INFO: Container webserver ready: true, restart count 0 Nov 23 16:15:30.003: INFO: cilium-q5tsk started at 2022-11-23 16:04:43 +0000 UTC (1+1 container statuses recorded) Nov 23 16:15:30.003: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:15:30.003: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:15:30.003: INFO: netserver-1 started at 2022-11-23 16:15:14 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:30.003: INFO: Container webserver ready: false, restart count 0 Nov 23 16:15:30.003: INFO: hostpath-client started at 2022-11-23 16:15:24 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:30.003: INFO: Container hostpath-client ready: true, restart count 0 Nov 23 16:15:30.003: INFO: ebs-csi-node-t56sd started at 2022-11-23 16:04:43 +0000 UTC (0+3 container statuses recorded) Nov 23 16:15:30.003: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:15:30.003: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:15:30.003: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:15:30.003: INFO: pod-qos-class-ef170dd3-fd69-418f-ac17-4fb3aedb39a6 started at 2022-11-23 16:13:39 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:30.003: INFO: Container agnhost ready: false, restart count 0 Nov 23 16:15:30.003: INFO: csi-hostpathplugin-0 started at 2022-11-23 16:15:03 +0000 UTC (0+7 container statuses recorded) Nov 23 16:15:30.003: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:15:30.003: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:15:30.003: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:15:30.003: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 23 16:15:30.003: INFO: Container hostpath ready: true, restart count 0 Nov 23 16:15:30.003: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:15:30.003: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:15:30.333: INFO: Latency metrics for node i-0ade3c3082581cb52 Nov 23 16:15:30.333: INFO: Logging node info for node i-0bf2756d84bae9168 Nov 23 16:15:30.395: INFO: Node Info: &Node{ObjectMeta:{i-0bf2756d84bae9168 2f26f127-5eb0-44f3-97f0-990045e7b900 18137 0 2022-11-23 16:04:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0bf2756d84bae9168 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0bf2756d84bae9168 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.218 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0bf2756d84bae9168"} io.cilium.network.ipv4-cilium-host:172.20.35.201 io.cilium.network.ipv4-health-ip:172.20.51.170 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:02 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:14:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0bf2756d84bae9168,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:02 +0000 UTC,LastTransitionTime:2022-11-23 16:05:02 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:14:46 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:14:46 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:14:46 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:14:46 +0000 UTC,LastTransitionTime:2022-11-23 16:04:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.218,},NodeAddress{Type:ExternalIP,Address:34.221.37.130,},NodeAddress{Type:InternalDNS,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-37-130.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d7db7f0ff759b7a6209e832f1dcea,SystemUUID:ec2d7db7-f0ff-759b-7a62-09e832f1dcea,BootID:8f294a85-a1ed-4dfb-bdb9-f8818e1f067f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f registry.k8s.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:15:30.395: INFO: Logging kubelet events for node i-0bf2756d84bae9168 Nov 23 16:15:30.459: INFO: Logging pods the kubelet thinks is on node i-0bf2756d84bae9168 Nov 23 16:15:30.525: INFO: cilium-scm24 started at 2022-11-23 16:04:35 +0000 UTC (1+1 container statuses recorded) Nov 23 16:15:30.525: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:15:30.525: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:15:30.525: INFO: ebs-csi-controller-847c4b8bc4-kwpsb started at 2022-11-23 16:04:56 +0000 UTC (0+5 container statuses recorded) Nov 23 16:15:30.525: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:15:30.525: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:15:30.525: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:15:30.525: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:15:30.525: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:15:30.525: INFO: pod-init-b2b9dfba-4d91-4037-9a97-0d324ebc8786 started at 2022-11-23 16:15:27 +0000 UTC (2+1 container statuses recorded) Nov 23 16:15:30.525: INFO: Init container init1 ready: true, restart count 0 Nov 23 16:15:30.525: INFO: Init container init2 ready: false, restart count 0 Nov 23 16:15:30.525: INFO: Container run1 ready: false, restart count 0 Nov 23 16:15:30.525: INFO: echo-sourceip started at 2022-11-23 16:15:27 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:30.525: INFO: Container agnhost-container ready: false, restart count 0 Nov 23 16:15:30.525: INFO: netserver-2 started at 2022-11-23 16:15:14 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:30.525: INFO: Container webserver ready: false, restart count 0 Nov 23 16:15:30.525: INFO: ebs-csi-node-crshz started at 2022-11-23 16:04:35 +0000 UTC (0+3 container statuses recorded) Nov 23 16:15:30.525: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:15:30.525: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:15:30.525: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:15:30.772: INFO: Latency metrics for node i-0bf2756d84bae9168 Nov 23 16:15:30.772: INFO: Logging node info for node i-0fe7a5031fb82cc5b Nov 23 16:15:30.834: INFO: Node Info: &Node{ObjectMeta:{i-0fe7a5031fb82cc5b 0297efe3-5070-45f1-8aa2-9a1056b17b59 17585 0 2022-11-23 16:04:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0fe7a5031fb82cc5b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0fe7a5031fb82cc5b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.89 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0fe7a5031fb82cc5b"} io.cilium.network.ipv4-cilium-host:172.20.57.57 io.cilium.network.ipv4-health-ip:172.20.59.99 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-11-23 16:04:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:03 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:14:38 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0fe7a5031fb82cc5b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:03 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:14:26 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:14:26 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:14:26 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:14:26 +0000 UTC,LastTransitionTime:2022-11-23 16:04:54 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.89,},NodeAddress{Type:ExternalIP,Address:34.221.99.167,},NodeAddress{Type:InternalDNS,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-99-167.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29292cb2c053c780af3b39ea2fa8c6,SystemUUID:ec29292c-b2c0-53c7-80af-3b39ea2fa8c6,BootID:ff4df73e-ccb2-4109-99ca-04a1b7ab5611,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:15:30.835: INFO: Logging kubelet events for node i-0fe7a5031fb82cc5b Nov 23 16:15:30.899: INFO: Logging pods the kubelet thinks is on node i-0fe7a5031fb82cc5b Nov 23 16:15:30.968: INFO: pod-exec-websocket-03f5b5a7-8bd9-4eee-81dd-4752aa7625f2 started at 2022-11-23 16:15:17 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:30.968: INFO: Container main ready: true, restart count 0 Nov 23 16:15:30.968: INFO: netserver-3 started at 2022-11-23 16:15:14 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:30.968: INFO: Container webserver ready: false, restart count 0 Nov 23 16:15:30.968: INFO: pod-terminate-status-2-12 started at 2022-11-23 16:15:30 +0000 UTC (1+1 container statuses recorded) Nov 23 16:15:30.968: INFO: Init container fail ready: false, restart count 0 Nov 23 16:15:30.968: INFO: Container blocked ready: false, restart count 0 Nov 23 16:15:30.968: INFO: ebs-csi-node-8ffwl started at 2022-11-23 16:04:34 +0000 UTC (0+3 container statuses recorded) Nov 23 16:15:30.968: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:15:30.968: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:15:30.968: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:15:30.968: INFO: coredns-autoscaler-db7b97744-r4jm7 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:30.968: INFO: Container autoscaler ready: true, restart count 0 Nov 23 16:15:30.968: INFO: pod-terminate-status-0-10 started at 2022-11-23 16:15:28 +0000 UTC (1+1 container statuses recorded) Nov 23 16:15:30.968: INFO: Init container fail ready: false, restart count 0 Nov 23 16:15:30.968: INFO: Container blocked ready: false, restart count 0 Nov 23 16:15:30.968: INFO: startup-e67fcc3c-55e9-42d3-b503-397a973eb07d started at 2022-11-23 16:14:41 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:30.968: INFO: Container busybox ready: true, restart count 0 Nov 23 16:15:30.968: INFO: cilium-727pv started at 2022-11-23 16:04:34 +0000 UTC (1+1 container statuses recorded) Nov 23 16:15:30.968: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:15:30.968: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:15:30.968: INFO: ebs-csi-controller-847c4b8bc4-lxdkr started at 2022-11-23 16:04:54 +0000 UTC (0+5 container statuses recorded) Nov 23 16:15:30.968: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:15:30.968: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:15:30.968: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:15:30.968: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:15:30.968: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:15:30.968: INFO: pod-terminate-status-1-10 started at 2022-11-23 16:15:28 +0000 UTC (1+1 container statuses recorded) Nov 23 16:15:30.968: INFO: Init container fail ready: false, restart count 0 Nov 23 16:15:30.968: INFO: Container blocked ready: false, restart count 0 Nov 23 16:15:30.968: INFO: coredns-6c7bddbb75-bpdc6 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded) Nov 23 16:15:30.968: INFO: Container coredns ready: true, restart count 0 Nov 23 16:15:31.324: INFO: Latency metrics for node i-0fe7a5031fb82cc5b Nov 23 16:15:31.324: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP:�[0m Destroying namespace "volume-expand-465" for this suite. �[38;5;243m11/23/22 16:15:31.387�[0m
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(block\svolmode\)\]\svolumeMode\sshould\snot\smount\s\/\smap\sunused\svolumes\sin\sa\spod\s\[LinuxOnly\]$'
test/e2e/storage/testsuites/volumemode.go:387 k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeModeTestSuite).DefineTests.func7() test/e2e/storage/testsuites/volumemode.go:387 +0x62bfrom junit_01.xml
{"msg":"FAILED External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","completed":10,"skipped":120,"failed":2,"failures":["External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]} [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode test/e2e/framework/framework.go:186 �[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m11/23/22 16:17:22.435�[0m Nov 23 16:17:22.435: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP:�[0m Building a namespace api object, basename volumemode �[38;5;243m11/23/22 16:17:22.436�[0m �[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m11/23/22 16:17:22.63�[0m �[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m11/23/22 16:17:22.756�[0m [It] should not mount / map unused volumes in a pod [LinuxOnly] test/e2e/storage/testsuites/volumemode.go:354 Nov 23 16:17:22.882: INFO: Creating resource for dynamic PV Nov 23 16:17:22.882: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP:�[0m creating a StorageClass volumemode-8642-e2e-sc4zd6b �[38;5;243m11/23/22 16:17:22.883�[0m �[1mSTEP:�[0m creating a claim �[38;5;243m11/23/22 16:17:22.947�[0m �[1mSTEP:�[0m Creating pod �[38;5;243m11/23/22 16:17:23.085�[0m Nov 23 16:17:23.151: INFO: Waiting up to 5m0s for pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc" in namespace "volumemode-8642" to be "running" Nov 23 16:17:23.214: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 63.269014ms Nov 23 16:17:25.281: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2.130057701s Nov 23 16:17:27.280: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4.128912179s Nov 23 16:17:29.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 6.127757122s Nov 23 16:17:31.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 8.127803325s Nov 23 16:17:33.296: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 10.145305462s Nov 23 16:17:35.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 12.12748912s Nov 23 16:17:37.280: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 14.129055724s Nov 23 16:17:39.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 16.127559214s Nov 23 16:17:41.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 18.12693318s Nov 23 16:17:43.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 20.127023119s Nov 23 16:17:45.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 22.126945354s Nov 23 16:17:47.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 24.126713265s Nov 23 16:17:49.287: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 26.135657548s Nov 23 16:17:51.283: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 28.132287317s Nov 23 16:17:53.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 30.127375786s Nov 23 16:17:55.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 32.127348062s Nov 23 16:17:57.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 34.127838018s Nov 23 16:17:59.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 36.127934135s Nov 23 16:18:01.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 38.126634723s Nov 23 16:18:03.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 40.127847002s Nov 23 16:18:05.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 42.127977565s Nov 23 16:18:07.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 44.127771152s Nov 23 16:18:09.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 46.127607111s Nov 23 16:18:11.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 48.127639373s Nov 23 16:18:13.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 50.128166783s Nov 23 16:18:15.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 52.127098989s Nov 23 16:18:17.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 54.127015238s Nov 23 16:18:19.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 56.127603132s Nov 23 16:18:21.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 58.127614768s Nov 23 16:18:23.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.127504501s Nov 23 16:18:25.285: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.134338854s Nov 23 16:18:27.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.128602177s Nov 23 16:18:29.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.126821299s Nov 23 16:18:31.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.127071894s Nov 23 16:18:33.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.127505945s Nov 23 16:18:35.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.127336671s Nov 23 16:18:37.282: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.131338436s Nov 23 16:18:39.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.127420412s Nov 23 16:18:41.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.128243514s Nov 23 16:18:43.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.127794607s Nov 23 16:18:45.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.127865713s Nov 23 16:18:47.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.127433738s Nov 23 16:18:49.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.127246198s Nov 23 16:18:51.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.127584976s Nov 23 16:18:53.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.127424515s Nov 23 16:18:55.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.127082934s Nov 23 16:18:57.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.128617254s Nov 23 16:18:59.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.127417998s Nov 23 16:19:01.283: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m38.131834283s Nov 23 16:19:03.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m40.127528905s Nov 23 16:19:05.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m42.127238815s Nov 23 16:19:07.298: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.147548133s Nov 23 16:19:09.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.128404727s Nov 23 16:19:11.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.126565826s Nov 23 16:19:13.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m50.127380453s Nov 23 16:19:15.333: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m52.182125622s Nov 23 16:19:17.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m54.127127043s Nov 23 16:19:19.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m56.12750057s Nov 23 16:19:21.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m58.127379447s Nov 23 16:19:23.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m0.12814135s Nov 23 16:19:25.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m2.127456222s Nov 23 16:19:27.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m4.128025391s Nov 23 16:19:29.280: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.129520403s Nov 23 16:19:31.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.12721616s Nov 23 16:19:33.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.126837821s Nov 23 16:19:35.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.126431033s Nov 23 16:19:37.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.127414458s Nov 23 16:19:39.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.127331013s Nov 23 16:19:41.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.126876404s Nov 23 16:19:43.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.12686167s Nov 23 16:19:45.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.127711796s Nov 23 16:19:47.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.127562344s Nov 23 16:19:49.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.126631081s Nov 23 16:19:51.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.127310413s Nov 23 16:19:53.283: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.132378783s Nov 23 16:19:55.280: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.129398005s Nov 23 16:19:57.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.127340435s Nov 23 16:19:59.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.128524511s Nov 23 16:20:01.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.127259703s Nov 23 16:20:03.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.126728652s Nov 23 16:20:05.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.127082471s Nov 23 16:20:07.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.127897456s Nov 23 16:20:09.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.127147851s Nov 23 16:20:11.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.127175501s Nov 23 16:20:13.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.126800977s Nov 23 16:20:15.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.127539664s Nov 23 16:20:17.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.126793694s Nov 23 16:20:19.280: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.128990473s Nov 23 16:20:21.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m58.126698627s Nov 23 16:20:23.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m0.127517543s Nov 23 16:20:25.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m2.127119199s Nov 23 16:20:27.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.127598618s Nov 23 16:20:29.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m6.127807334s Nov 23 16:20:31.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m8.127519357s Nov 23 16:20:33.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.127147123s Nov 23 16:20:35.281: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.130443325s Nov 23 16:20:37.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m14.126714487s Nov 23 16:20:39.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m16.12794512s Nov 23 16:20:41.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m18.128239244s Nov 23 16:20:43.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m20.126734795s Nov 23 16:20:45.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m22.127386392s Nov 23 16:20:47.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.127059529s Nov 23 16:20:49.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.126812281s Nov 23 16:20:51.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.126943732s Nov 23 16:20:53.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.126251507s Nov 23 16:20:55.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.126879504s Nov 23 16:20:57.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m34.126308988s Nov 23 16:20:59.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m36.126192929s Nov 23 16:21:01.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m38.126870895s Nov 23 16:21:03.281: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m40.130519227s Nov 23 16:21:05.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m42.126370152s Nov 23 16:21:07.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m44.126641486s Nov 23 16:21:09.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m46.126786807s Nov 23 16:21:11.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m48.126514432s Nov 23 16:21:13.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m50.126584411s Nov 23 16:21:15.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m52.126617381s Nov 23 16:21:17.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m54.12618347s Nov 23 16:21:19.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m56.12656001s Nov 23 16:21:21.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m58.127629001s Nov 23 16:21:23.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m0.126899209s Nov 23 16:21:25.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m2.125949764s Nov 23 16:21:27.280: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m4.12951611s Nov 23 16:21:29.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.126108055s Nov 23 16:21:31.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.126647695s Nov 23 16:21:33.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.126109078s Nov 23 16:21:35.281: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.129748862s Nov 23 16:21:37.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.126848165s Nov 23 16:21:39.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m16.126375406s Nov 23 16:21:41.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m18.126772759s Nov 23 16:21:43.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m20.126423955s Nov 23 16:21:45.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m22.126698223s Nov 23 16:21:47.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m24.126048036s Nov 23 16:21:49.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.126181821s Nov 23 16:21:51.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.127033431s Nov 23 16:21:53.280: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.128783891s Nov 23 16:21:55.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.126509756s Nov 23 16:21:57.279: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.128599332s Nov 23 16:21:59.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.126002357s Nov 23 16:22:01.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.12636139s Nov 23 16:22:03.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.12658374s Nov 23 16:22:05.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.126519371s Nov 23 16:22:07.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.127149329s Nov 23 16:22:09.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.126405701s Nov 23 16:22:11.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.126454315s Nov 23 16:22:13.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.12698304s Nov 23 16:22:15.277: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.126425314s Nov 23 16:22:17.281: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.13013359s Nov 23 16:22:19.280: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.128888869s Nov 23 16:22:21.287: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.135889502s Nov 23 16:22:23.278: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.126740815s Nov 23 16:22:23.340: INFO: Pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.189407783s Nov 23 16:22:23.341: INFO: Unexpected error: <*pod.timeoutError | 0xc003ff27e0>: { msg: "timed out while waiting for pod volumemode-8642/pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc to be running", observedObjects: [ <*v1.Pod | 0xc001631000>{ TypeMeta: {Kind: "", APIVersion: ""}, ObjectMeta: { Name: "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc", GenerateName: "", Namespace: "volumemode-8642", SelfLink: "", UID: "1610a6e3-c6b5-4c4a-9a45-a3bd4f0a0b4d", ResourceVersion: "22242", Generation: 0, CreationTimestamp: { Time: { wall: 0, ext: 63804817043, loc: { name: "Local", zone: [ {name: "UTC", offset: 0, isDST: false}, ], tx: [ { when: -576460752303423488, index: 0, isstd: false, isutc: false, }, ], extend: "UTC0", cacheStart: 9223372036854775807, cacheEnd: 9223372036854775807, cacheZone: {name: "UTC", offset: 0, isDST: false}, }, }, }, DeletionTimestamp: nil, DeletionGracePeriodSeconds: nil, Labels: nil, Annotations: nil, OwnerReferences: nil, Finalizers: nil, ManagedFields: [ { Manager: "e2e.test", Operation: "Update", APIVersion: "v1", Time: { Time: { wall: 0, ext: 63804817043, loc: { name: "Local", zone: [...], tx: [...], extend: "UTC0", cacheStart: 9223372036854775807, cacheEnd: 9223372036854775807, cacheZone: {name: ..., offset: ..., isDST: ...}, }, }, }, FieldsType: "FieldsV1", FieldsV1: { Raw: "{\"f:spec\":{\"f:containers\":{\"k:{\\\"name\\\":\\\"write-pod\\\"}\":{\".\":{},\"f:command\":{},\"f:image\":{},\"f:imagePullPolicy\":{},\"f:name\":{},\"f:resources\":{},\"f:securityContext\":{\".\":{},\"f:privileged\":{}},\"f:terminationMessagePath\":{},\"f:terminationMessagePolicy\":{}}},\"f:dnsPolicy\":{},\"f:enableServiceLinks\":{},\"f:restartPolicy\":{},\"f:schedulerName\":{},\"f:securityContext\":{\".\":{},\"f:fsGroup\":{},\"f:seLinuxOptions\":{\".\":{},\"f:level\":{}}},\"f:terminationGracePeriodSeconds\":{},\"f:volumes\":{\".\":{},\"k:{\\\"name\\\":\\\"volume1\\\"}\":{\".\":{},\"f:name\":{},\"f:persistentVolumeClaim\":{\".\":{},\"f:claimName\":{}}}}}}", }, Subresource: "", }, ], }, Spec: { Volumes: [ { Name: "vol... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output Nov 23 16:22:23.341: FAIL: timed out while waiting for pod volumemode-8642/pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc to be running Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeModeTestSuite).DefineTests.func7() test/e2e/storage/testsuites/volumemode.go:387 +0x62b Nov 23 16:22:23.341: INFO: Deleting pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc" in namespace "volumemode-8642" Nov 23 16:22:23.410: INFO: Wait up to 5m0s for pod "pod-7a83e71f-6638-4cb2-8477-f0b8a7d47edc" to be fully deleted �[1mSTEP:�[0m Deleting pvc �[38;5;243m11/23/22 16:22:23.472�[0m Nov 23 16:22:23.534: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comh4w8p" �[1mSTEP:�[0m Deleting sc �[38;5;243m11/23/22 16:22:23.597�[0m [AfterEach] [Testpattern: Dynamic PV (block volmode)] volumeMode test/e2e/framework/framework.go:187 �[1mSTEP:�[0m Collecting events from namespace "volumemode-8642". �[38;5;243m11/23/22 16:22:23.661�[0m �[1mSTEP:�[0m Found 5 events. �[38;5;243m11/23/22 16:22:23.725�[0m Nov 23 16:22:23.725: INFO: At 2022-11-23 16:17:22 +0000 UTC - event for ebs.csi.aws.comh4w8p: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 16:22:23.725: INFO: At 2022-11-23 16:17:23 +0000 UTC - event for ebs.csi.aws.comh4w8p: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 16:22:23.725: INFO: At 2022-11-23 16:17:23 +0000 UTC - event for ebs.csi.aws.comh4w8p: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } Provisioning: External provisioner is provisioning volume for claim "volumemode-8642/ebs.csi.aws.comh4w8p" Nov 23 16:22:23.725: INFO: At 2022-11-23 16:17:33 +0000 UTC - event for ebs.csi.aws.comh4w8p: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "volumemode-8642-e2e-sc4zd6b": rpc error: code = Internal desc = Could not create volume "pvc-561ee841-0a9f-42f1-9bc7-4200d9702cd6": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Nov 23 16:22:23.725: INFO: At 2022-11-23 16:17:44 +0000 UTC - event for ebs.csi.aws.comh4w8p: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "volumemode-8642-e2e-sc4zd6b": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 16:22:23.787: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 16:22:23.787: INFO: Nov 23 16:22:23.851: INFO: Logging node info for node i-060d86c6ffc5622b0 Nov 23 16:22:23.914: INFO: Node Info: &Node{ObjectMeta:{i-060d86c6ffc5622b0 b49ff0b3-8686-4c0f-b991-1d427ca43e2a 27298 0 2022-11-23 16:02:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-060d86c6ffc5622b0 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.40.57 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-060d86c6ffc5622b0"} io.cilium.network.ipv4-cilium-host:172.20.57.204 io.cilium.network.ipv4-health-ip:172.20.37.200 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 16:02:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2022-11-23 16:03:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:03:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-060d86c6ffc5622b0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:04:42 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:03:24 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.57,},NodeAddress{Type:ExternalIP,Address:54.200.161.23,},NodeAddress{Type:InternalDNS,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-200-161-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec226e0d77c347467223a292b33eb0c5,SystemUUID:ec226e0d-77c3-4746-7223-a292b33eb0c5,BootID:936b9dd2-17e7-49ac-8c1e-7556bf2c9ee8,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.25.4],SizeBytes:129085904,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.25.4],SizeBytes:118436153,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.25.4],SizeBytes:51921208,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42928334,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41477333,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:85d3f1e9dacc72531445989bb10999e1e70ebc409d11be57e5baa5f031a893b0 registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1],SizeBytes:18257577,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955428,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:22:23.914: INFO: Logging kubelet events for node i-060d86c6ffc5622b0 Nov 23 16:22:23.981: INFO: Logging pods the kubelet thinks is on node i-060d86c6ffc5622b0 Nov 23 16:22:24.049: INFO: etcd-manager-events-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.049: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 16:22:24.049: INFO: kops-controller-7w98n started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.049: INFO: Container kops-controller ready: true, restart count 0 Nov 23 16:22:24.049: INFO: dns-controller-89c6d5f67-9hm78 started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.049: INFO: Container dns-controller ready: true, restart count 0 Nov 23 16:22:24.049: INFO: cilium-f2gg7 started at 2022-11-23 16:03:06 +0000 UTC (1+1 container statuses recorded) Nov 23 16:22:24.049: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:22:24.049: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 16:22:24.049: INFO: cilium-operator-7f7bcdf787-xnz2q started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.049: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 16:22:24.049: INFO: aws-cloud-controller-manager-dzzkp started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.049: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Nov 23 16:22:24.049: INFO: etcd-manager-main-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.049: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 16:22:24.049: INFO: kube-apiserver-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+2 container statuses recorded) Nov 23 16:22:24.049: INFO: Container healthcheck ready: true, restart count 0 Nov 23 16:22:24.049: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 16:22:24.049: INFO: kube-controller-manager-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.049: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 16:22:24.049: INFO: kube-scheduler-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.049: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 16:22:24.049: INFO: ebs-csi-node-vpv4t started at 2022-11-23 16:03:06 +0000 UTC (0+3 container statuses recorded) Nov 23 16:22:24.049: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:22:24.049: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:22:24.049: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:22:24.315: INFO: Latency metrics for node i-060d86c6ffc5622b0 Nov 23 16:22:24.315: INFO: Logging node info for node i-0765e732a9635923b Nov 23 16:22:24.378: INFO: Node Info: &Node{ObjectMeta:{i-0765e732a9635923b 10981661-cc27-412a-9e66-7a37930b898f 28980 0 2022-11-23 16:04:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0765e732a9635923b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0765e732a9635923b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.58.74 csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9722":"i-0765e732a9635923b","ebs.csi.aws.com":"i-0765e732a9635923b"} io.cilium.network.ipv4-cilium-host:172.20.54.198 io.cilium.network.ipv4-health-ip:172.20.60.24 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:07 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:21:44 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0765e732a9635923b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:07 +0000 UTC,LastTransitionTime:2022-11-23 16:05:07 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:48 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:48 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:48 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:21:48 +0000 UTC,LastTransitionTime:2022-11-23 16:05:00 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.58.74,},NodeAddress{Type:ExternalIP,Address:52.25.125.23,},NodeAddress{Type:InternalDNS,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-25-125-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec23d129b4036d8d979053444dfae008,SystemUUID:ec23d129-b403-6d8d-9790-53444dfae008,BootID:fe390540-4601-456e-9686-93a36ea3dc8f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-9722^eb42a40e-6b4a-11ed-8178-a6469949b10b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-9722^eb42a40e-6b4a-11ed-8178-a6469949b10b,DevicePath:,},},Config:nil,},} Nov 23 16:22:24.378: INFO: Logging kubelet events for node i-0765e732a9635923b Nov 23 16:22:24.444: INFO: Logging pods the kubelet thinks is on node i-0765e732a9635923b Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-pbsv8 started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.517: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-rsgxx started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.517: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-dw4tg started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-s6nft started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.517: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-j452c started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-gfmtp started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-s6tqd started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-pkrml started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-jmvlq started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.517: INFO: cilium-czwdq started at 2022-11-23 16:04:40 +0000 UTC (1+1 container statuses recorded) Nov 23 16:22:24.517: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:22:24.517: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-87s2b started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.517: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 16:21:38 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.517: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-nghrb started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-jtsct started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.517: INFO: coredns-6c7bddbb75-2n2dw started at 2022-11-23 16:05:19 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.517: INFO: Container coredns ready: true, restart count 0 Nov 23 16:22:24.517: INFO: test-rolling-update-with-lb-dbbf5f9b8-hp5hf started at 2022-11-23 16:21:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.517: INFO: Container agnhost ready: true, restart count 0 Nov 23 16:22:24.517: INFO: deployment-97c6466db-trmfq started at 2022-11-23 16:20:52 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.517: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-mhzbk started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.517: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-9jhxg started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.517: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-4qdtt started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-bnm24 started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.517: INFO: pvc-volume-tester-fb7cs started at 2022-11-23 16:21:43 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.517: INFO: Container volume-tester ready: true, restart count 0 Nov 23 16:22:24.517: INFO: netserver-0 started at 2022-11-23 16:22:03 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.517: INFO: Container webserver ready: false, restart count 0 Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-5l98l started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.517: INFO: simpletest-rc-to-be-deleted-npl9p started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.517: INFO: hostexec-i-0765e732a9635923b-mbrlj started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.518: INFO: ebs-csi-node-5khlc started at 2022-11-23 16:04:40 +0000 UTC (0+3 container statuses recorded) Nov 23 16:22:24.518: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:22:24.518: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:22:24.518: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:22:24.518: INFO: simpletest-rc-to-be-deleted-4sztl started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.518: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:24.518: INFO: simpletest-rc-to-be-deleted-mz7gg started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.518: INFO: simpletest-rc-to-be-deleted-l7l9k started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:24.518: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:24.518: INFO: simpletest-rc-to-be-deleted-ht4k5 started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.518: INFO: csi-mockplugin-0 started at 2022-11-23 16:21:38 +0000 UTC (0+3 container statuses recorded) Nov 23 16:22:24.518: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:22:24.518: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 16:22:24.518: INFO: Container mock ready: true, restart count 0 Nov 23 16:22:24.518: INFO: simpletest-rc-to-be-deleted-9nsjr started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.518: INFO: simpletest-rc-to-be-deleted-ksbhs started at <nil> (0+0 container statuses recorded) Nov 23 16:22:24.518: INFO: simpletest-rc-to-be-deleted-7kw6r started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.103: INFO: Latency metrics for node i-0765e732a9635923b Nov 23 16:22:25.103: INFO: Logging node info for node i-0ade3c3082581cb52 Nov 23 16:22:25.165: INFO: Node Info: &Node{ObjectMeta:{i-0ade3c3082581cb52 2305662a-7d05-4979-bcec-220b8b4672db 29101 0 2022-11-23 16:04:42 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0ade3c3082581cb52 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0ade3c3082581cb52 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.56.12 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-4466":"i-0ade3c3082581cb52","ebs.csi.aws.com":"i-0ade3c3082581cb52"} io.cilium.network.ipv4-cilium-host:172.20.45.237 io.cilium.network.ipv4-health-ip:172.20.55.90 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:21:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0ade3c3082581cb52,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:12 +0000 UTC,LastTransitionTime:2022-11-23 16:05:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:21 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:21 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:21 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:21:21 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.56.12,},NodeAddress{Type:ExternalIP,Address:35.92.243.169,},NodeAddress{Type:InternalDNS,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-92-243-169.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bd980dfb52643cb4857d57b45824b,SystemUUID:ec2bd980-dfb5-2643-cb48-57d57b45824b,BootID:55912341-dce8-4199-a57c-c8703e24af46,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 registry.k8s.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:22:25.166: INFO: Logging kubelet events for node i-0ade3c3082581cb52 Nov 23 16:22:25.232: INFO: Logging pods the kubelet thinks is on node i-0ade3c3082581cb52 Nov 23 16:22:25.303: INFO: csi-hostpathplugin-0 started at 2022-11-23 16:21:50 +0000 UTC (0+7 container statuses recorded) Nov 23 16:22:25.303: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:22:25.303: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:22:25.303: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:22:25.303: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 23 16:22:25.303: INFO: Container hostpath ready: true, restart count 0 Nov 23 16:22:25.303: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:22:25.303: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:22:25.303: INFO: simpletest-rc-to-be-deleted-qhjhq started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.303: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.303: INFO: simpletest-rc-to-be-deleted-r9jmt started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-4fh26 started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-hzrps started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.304: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.304: INFO: inline-volume-tester-dc5fg started at 2022-11-23 16:21:50 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.304: INFO: Container csi-volume-tester ready: true, restart count 0 Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-pwb9n started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-pcffw started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: netserver-1 started at 2022-11-23 16:22:03 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.304: INFO: Container webserver ready: false, restart count 0 Nov 23 16:22:25.304: INFO: test-rolling-update-with-lb-dbbf5f9b8-bhfzj started at 2022-11-23 16:21:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.304: INFO: Container agnhost ready: true, restart count 0 Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-dgcmw started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.304: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-ph7f2 started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-4wrsz started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: cilium-q5tsk started at 2022-11-23 16:04:43 +0000 UTC (1+1 container statuses recorded) Nov 23 16:22:25.304: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:22:25.304: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-fnnf7 started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.304: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-lvwg6 started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-k9974 started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-zzwqq started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-n7vjs started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.304: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-hckpk started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: ebs-csi-node-t56sd started at 2022-11-23 16:04:43 +0000 UTC (0+3 container statuses recorded) Nov 23 16:22:25.304: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:22:25.304: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:22:25.304: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-gjt6m started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.304: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-xcm5t started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.304: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-flqw4 started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-b4tvd started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-zhxwp started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-4dqbb started at 2022-11-23 16:22:19 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.304: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-r7qcw started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-8289m started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-vldhq started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-9l5rw started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.304: INFO: simpletest-rc-to-be-deleted-fbwc2 started at 2022-11-23 16:22:19 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.304: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.697: INFO: Latency metrics for node i-0ade3c3082581cb52 Nov 23 16:22:25.697: INFO: Logging node info for node i-0bf2756d84bae9168 Nov 23 16:22:25.760: INFO: Node Info: &Node{ObjectMeta:{i-0bf2756d84bae9168 2f26f127-5eb0-44f3-97f0-990045e7b900 27933 0 2022-11-23 16:04:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0bf2756d84bae9168 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0bf2756d84bae9168 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.218 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0bf2756d84bae9168"} io.cilium.network.ipv4-cilium-host:172.20.35.201 io.cilium.network.ipv4-health-ip:172.20.51.170 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:02 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:20:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0bf2756d84bae9168,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:02 +0000 UTC,LastTransitionTime:2022-11-23 16:05:02 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.218,},NodeAddress{Type:ExternalIP,Address:34.221.37.130,},NodeAddress{Type:InternalDNS,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-37-130.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d7db7f0ff759b7a6209e832f1dcea,SystemUUID:ec2d7db7-f0ff-759b-7a62-09e832f1dcea,BootID:8f294a85-a1ed-4dfb-bdb9-f8818e1f067f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f registry.k8s.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:22:25.760: INFO: Logging kubelet events for node i-0bf2756d84bae9168 Nov 23 16:22:25.831: INFO: Logging pods the kubelet thinks is on node i-0bf2756d84bae9168 Nov 23 16:22:25.902: INFO: ebs-csi-node-crshz started at 2022-11-23 16:04:35 +0000 UTC (0+3 container statuses recorded) Nov 23 16:22:25.902: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:22:25.902: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:22:25.902: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-2sfng started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-qnxzc started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-m6n9w started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.902: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-8r4rr started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-qrpvs started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-m29cj started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: inline-volume-nvmnf started at 2022-11-23 16:21:52 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.902: INFO: Container volume-tester ready: false, restart count 0 Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-57xqj started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.902: INFO: Container nginx ready: true, restart count 0 Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-5fq4k started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.902: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-l467n started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.902: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-p8wmc started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-mv626 started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-2dtgh started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-brxq6 started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: ebs-csi-controller-847c4b8bc4-kwpsb started at 2022-11-23 16:04:56 +0000 UTC (0+5 container statuses recorded) Nov 23 16:22:25.902: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:22:25.902: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:22:25.902: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:22:25.902: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:22:25.902: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:22:25.902: INFO: sample-webhook-deployment-5d85dd8cdb-gvmr5 started at 2022-11-23 16:22:17 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.902: INFO: Container sample-webhook ready: false, restart count 0 Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-p2wj2 started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.902: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-pm9xt started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.902: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-4bhmj started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-b9vrl started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-jt8d7 started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-pq96j started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-krr4f started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-665kq started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: cilium-scm24 started at 2022-11-23 16:04:35 +0000 UTC (1+1 container statuses recorded) Nov 23 16:22:25.902: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:22:25.902: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:22:25.902: INFO: test-rolling-update-with-lb-dbbf5f9b8-7fpt7 started at 2022-11-23 16:21:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.902: INFO: Container agnhost ready: true, restart count 0 Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-qrpk6 started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.902: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-vlbnd started at <nil> (0+0 container statuses recorded) Nov 23 16:22:25.902: INFO: netserver-2 started at 2022-11-23 16:22:03 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.902: INFO: Container webserver ready: false, restart count 0 Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-wjvpl started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:25.902: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:25.902: INFO: simpletest-rc-to-be-deleted-qqqsz started at <nil> (0+0 container statuses recorded) Nov 23 16:22:26.279: INFO: Latency metrics for node i-0bf2756d84bae9168 Nov 23 16:22:26.279: INFO: Logging node info for node i-0fe7a5031fb82cc5b Nov 23 16:22:26.342: INFO: Node Info: &Node{ObjectMeta:{i-0fe7a5031fb82cc5b 0297efe3-5070-45f1-8aa2-9a1056b17b59 29165 0 2022-11-23 16:04:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0fe7a5031fb82cc5b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0fe7a5031fb82cc5b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.89 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-930":"i-0fe7a5031fb82cc5b","ebs.csi.aws.com":"i-0fe7a5031fb82cc5b"} io.cilium.network.ipv4-cilium-host:172.20.57.57 io.cilium.network.ipv4-health-ip:172.20.59.99 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-11-23 16:04:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:03 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 16:21:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0fe7a5031fb82cc5b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:03 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:21:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:54 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.89,},NodeAddress{Type:ExternalIP,Address:34.221.99.167,},NodeAddress{Type:InternalDNS,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-99-167.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29292cb2c053c780af3b39ea2fa8c6,SystemUUID:ec29292c-b2c0-53c7-80af-3b39ea2fa8c6,BootID:ff4df73e-ccb2-4109-99ca-04a1b7ab5611,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-930^ed65e6a3-6b4a-11ed-9ffc-1aa129783697],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-930^ed65e6a3-6b4a-11ed-9ffc-1aa129783697,DevicePath:,},},Config:nil,},} Nov 23 16:22:26.342: INFO: Logging kubelet events for node i-0fe7a5031fb82cc5b Nov 23 16:22:26.407: INFO: Logging pods the kubelet thinks is on node i-0fe7a5031fb82cc5b Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-5jbr4 started at <nil> (0+0 container statuses recorded) Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-kghcj started at <nil> (0+0 container statuses recorded) Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-5w4sf started at <nil> (0+0 container statuses recorded) Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-8mffh started at <nil> (0+0 container statuses recorded) Nov 23 16:22:26.483: INFO: ebs-csi-node-8ffwl started at 2022-11-23 16:04:34 +0000 UTC (0+3 container statuses recorded) Nov 23 16:22:26.483: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:22:26.483: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:22:26.483: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-9nz8s started at <nil> (0+0 container statuses recorded) Nov 23 16:22:26.483: INFO: cilium-727pv started at 2022-11-23 16:04:34 +0000 UTC (1+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:22:26.483: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-kwv6j started at <nil> (0+0 container statuses recorded) Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-xvmkq started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-qdf9s started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-ckkht started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-mltd5 started at 2022-11-23 16:22:19 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-bl4p7 started at <nil> (0+0 container statuses recorded) Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-qfd7r started at 2022-11-23 16:22:19 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:26.483: INFO: coredns-6c7bddbb75-bpdc6 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container coredns ready: true, restart count 0 Nov 23 16:22:26.483: INFO: netserver-3 started at 2022-11-23 16:22:03 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container webserver ready: false, restart count 0 Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-4gx62 started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-bg29h started at 2022-11-23 16:22:19 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-8b94s started at <nil> (0+0 container statuses recorded) Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-l9vd2 started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-c4tmw started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-52ltw started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-hzl5p started at 2022-11-23 16:22:18 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-4c77d started at 2022-11-23 16:22:21 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:26.483: INFO: coredns-autoscaler-db7b97744-r4jm7 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container autoscaler ready: true, restart count 0 Nov 23 16:22:26.483: INFO: inline-volume-tester-dvvp2 started at 2022-11-23 16:21:48 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container csi-volume-tester ready: true, restart count 0 Nov 23 16:22:26.483: INFO: ebs-csi-controller-847c4b8bc4-lxdkr started at 2022-11-23 16:04:54 +0000 UTC (0+5 container statuses recorded) Nov 23 16:22:26.483: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:22:26.483: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:22:26.483: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:22:26.483: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:22:26.483: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-zjq79 started at <nil> (0+0 container statuses recorded) Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-44rdq started at 2022-11-23 16:22:20 +0000 UTC (0+1 container statuses recorded) Nov 23 16:22:26.483: INFO: Container nginx ready: false, restart count 0 Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-bcgnw started at <nil> (0+0 container statuses recorded) Nov 23 16:22:26.483: INFO: simpletest-rc-to-be-deleted-22l6n started at <nil> (0+0 container statuses recorded) Nov 23 16:22:26.483: INFO: csi-hostpathplugin-0 started at 2022-11-23 16:21:44 +0000 UTC (0+7 container statuses recorded) Nov 23 16:22:26.483: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:22:26.483: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:22:26.483: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:22:26.483: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 23 16:22:26.483: INFO: Container hostpath ready: true, restart count 0 Nov 23 16:22:26.483: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:22:26.483: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:22:27.301: INFO: Latency metrics for node i-0fe7a5031fb82cc5b Nov 23 16:22:27.301: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP:�[0m Destroying namespace "volumemode-8642" for this suite. �[38;5;243m11/23/22 16:22:27.365�[0m
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(block\svolmode\)\]\svolumes\sshould\sstore\sdata$'
test/e2e/storage/testsuites/volumes.go:188 k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumesTestSuite).DefineTests.func3() test/e2e/storage/testsuites/volumes.go:188 +0x457from junit_01.xml
{"msg":"FAILED External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (block volmode)] volumes should store data","completed":13,"skipped":102,"failed":2,"failures":["External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works","External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (block volmode)] volumes should store data"]} [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumes test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumes test/e2e/framework/framework.go:186 �[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m11/23/22 16:16:14.141�[0m Nov 23 16:16:14.141: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP:�[0m Building a namespace api object, basename volume �[38;5;243m11/23/22 16:16:14.142�[0m �[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m11/23/22 16:16:14.343�[0m �[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m11/23/22 16:16:14.474�[0m [It] should store data test/e2e/storage/testsuites/volumes.go:161 Nov 23 16:16:14.605: INFO: Creating resource for dynamic PV Nov 23 16:16:14.605: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP:�[0m creating a StorageClass volume-9370-e2e-scktnsj �[38;5;243m11/23/22 16:16:14.606�[0m �[1mSTEP:�[0m creating a claim �[38;5;243m11/23/22 16:16:14.673�[0m �[1mSTEP:�[0m starting external-injector �[38;5;243m11/23/22 16:16:14.809�[0m Nov 23 16:16:14.878: INFO: Waiting up to 5m0s for pod "external-injector" in namespace "volume-9370" to be "running" Nov 23 16:16:14.944: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 66.54047ms Nov 23 16:16:17.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2.133363105s Nov 23 16:16:19.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4.134138683s Nov 23 16:16:21.018: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 6.140654239s Nov 23 16:16:23.014: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 8.135746186s Nov 23 16:16:25.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 10.134041367s Nov 23 16:16:27.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 12.134736708s Nov 23 16:16:29.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 14.133744795s Nov 23 16:16:31.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 16.134192269s Nov 23 16:16:33.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 18.133835333s Nov 23 16:16:35.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 20.134217665s Nov 23 16:16:37.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 22.13317729s Nov 23 16:16:39.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 24.13369571s Nov 23 16:16:41.019: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 26.141269215s Nov 23 16:16:43.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 28.134442005s Nov 23 16:16:45.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 30.133709588s Nov 23 16:16:47.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 32.134159931s Nov 23 16:16:49.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 34.134377565s Nov 23 16:16:51.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 36.133861447s Nov 23 16:16:53.015: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 38.13724073s Nov 23 16:16:55.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 40.133435465s Nov 23 16:16:57.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 42.133395112s Nov 23 16:16:59.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 44.134373111s Nov 23 16:17:01.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 46.134517374s Nov 23 16:17:03.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 48.13411346s Nov 23 16:17:05.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 50.133881345s Nov 23 16:17:07.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 52.134437722s Nov 23 16:17:09.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 54.134739377s Nov 23 16:17:11.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 56.134968948s Nov 23 16:17:13.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 58.134429151s Nov 23 16:17:15.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.133970349s Nov 23 16:17:17.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.13494103s Nov 23 16:17:19.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.134001174s Nov 23 16:17:21.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.134496458s Nov 23 16:17:23.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.135015332s Nov 23 16:17:25.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.134335447s Nov 23 16:17:27.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.134157992s Nov 23 16:17:29.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.134732503s Nov 23 16:17:31.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.135658624s Nov 23 16:17:33.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.13442291s Nov 23 16:17:35.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.134184579s Nov 23 16:17:37.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.134689402s Nov 23 16:17:39.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.134600943s Nov 23 16:17:41.014: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.136095689s Nov 23 16:17:43.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.134078952s Nov 23 16:17:45.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.133329421s Nov 23 16:17:47.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.134222063s Nov 23 16:17:49.017: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.139714815s Nov 23 16:17:51.021: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.142968561s Nov 23 16:17:53.026: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m38.148634181s Nov 23 16:17:55.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m40.134370073s Nov 23 16:17:57.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m42.133587051s Nov 23 16:17:59.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.134153455s Nov 23 16:18:01.020: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.141970918s Nov 23 16:18:03.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.134506733s Nov 23 16:18:05.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m50.134382527s Nov 23 16:18:07.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m52.134146236s Nov 23 16:18:09.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m54.134896489s Nov 23 16:18:11.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m56.133728774s Nov 23 16:18:13.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 1m58.133778118s Nov 23 16:18:15.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m0.133609204s Nov 23 16:18:17.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m2.134502691s Nov 23 16:18:19.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m4.133511194s Nov 23 16:18:21.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.135022735s Nov 23 16:18:23.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.133747555s Nov 23 16:18:25.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.134423245s Nov 23 16:18:27.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.134177637s Nov 23 16:18:29.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.134401585s Nov 23 16:18:31.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.134071486s Nov 23 16:18:33.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.134045135s Nov 23 16:18:35.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.133893497s Nov 23 16:18:37.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.135029192s Nov 23 16:18:39.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.133884402s Nov 23 16:18:41.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.134555656s Nov 23 16:18:43.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.13385058s Nov 23 16:18:45.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.134440275s Nov 23 16:18:47.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.134210127s Nov 23 16:18:49.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.134157789s Nov 23 16:18:51.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.135076598s Nov 23 16:18:53.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.134032257s Nov 23 16:18:55.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.133891914s Nov 23 16:18:57.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.133843545s Nov 23 16:18:59.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.134332163s Nov 23 16:19:01.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.13447453s Nov 23 16:19:03.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.133498813s Nov 23 16:19:05.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.134580496s Nov 23 16:19:07.016: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.13836033s Nov 23 16:19:09.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.134025784s Nov 23 16:19:11.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.135272802s Nov 23 16:19:13.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 2m58.134589655s Nov 23 16:19:15.016: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m0.138630235s Nov 23 16:19:17.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m2.134626911s Nov 23 16:19:19.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.134325594s Nov 23 16:19:21.015: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m6.137486477s Nov 23 16:19:23.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m8.134222692s Nov 23 16:19:25.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.134386957s Nov 23 16:19:27.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.133889739s Nov 23 16:19:29.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m14.134328533s Nov 23 16:19:31.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m16.133577327s Nov 23 16:19:33.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m18.134379659s Nov 23 16:19:35.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m20.134342362s Nov 23 16:19:37.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m22.135092482s Nov 23 16:19:39.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.13395104s Nov 23 16:19:41.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.133721963s Nov 23 16:19:43.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.134821685s Nov 23 16:19:45.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.133202031s Nov 23 16:19:47.017: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.139184998s Nov 23 16:19:49.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m34.133980619s Nov 23 16:19:51.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m36.134571582s Nov 23 16:19:53.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m38.134285445s Nov 23 16:19:55.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m40.133802092s Nov 23 16:19:57.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m42.134471617s Nov 23 16:19:59.018: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m44.13999586s Nov 23 16:20:01.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m46.135595668s Nov 23 16:20:03.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m48.13512594s Nov 23 16:20:05.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m50.134624763s Nov 23 16:20:07.020: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m52.142137954s Nov 23 16:20:09.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m54.133796043s Nov 23 16:20:11.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m56.134886744s Nov 23 16:20:13.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 3m58.134620226s Nov 23 16:20:15.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m0.133549131s Nov 23 16:20:17.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m2.13565566s Nov 23 16:20:19.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m4.134266284s Nov 23 16:20:21.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.133907814s Nov 23 16:20:23.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.133999998s Nov 23 16:20:25.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.134275962s Nov 23 16:20:27.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.133915908s Nov 23 16:20:29.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.134636983s Nov 23 16:20:31.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m16.134516164s Nov 23 16:20:33.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m18.134315705s Nov 23 16:20:35.021: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m20.143196054s Nov 23 16:20:37.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m22.13545571s Nov 23 16:20:39.014: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m24.135795123s Nov 23 16:20:41.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.134421826s Nov 23 16:20:43.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.134239537s Nov 23 16:20:45.011: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.133606189s Nov 23 16:20:47.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.134415859s Nov 23 16:20:49.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.134436303s Nov 23 16:20:51.015: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.136814491s Nov 23 16:20:53.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.135044809s Nov 23 16:20:55.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.134944994s Nov 23 16:20:57.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.134715343s Nov 23 16:20:59.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.134796297s Nov 23 16:21:01.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.135270296s Nov 23 16:21:03.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.135599924s Nov 23 16:21:05.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.134075868s Nov 23 16:21:07.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.134623384s Nov 23 16:21:09.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.134488499s Nov 23 16:21:11.012: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.134709395s Nov 23 16:21:13.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.13474889s Nov 23 16:21:15.013: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.13511888s Nov 23 16:21:15.080: INFO: Pod "external-injector": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.201967631s �[1mSTEP:�[0m Deleting pod external-injector in namespace volume-9370 �[38;5;243m11/23/22 16:21:15.08�[0m Nov 23 16:21:15.154: INFO: Waiting for pod external-injector to disappear Nov 23 16:21:15.221: INFO: Pod external-injector no longer exists Nov 23 16:21:15.221: FAIL: Failed to create injector pod: timed out while waiting for pod volume-9370/external-injector to be running Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumesTestSuite).DefineTests.func3() test/e2e/storage/testsuites/volumes.go:188 +0x457 �[1mSTEP:�[0m cleaning the environment after external �[38;5;243m11/23/22 16:21:15.221�[0m �[1mSTEP:�[0m Deleting pvc �[38;5;243m11/23/22 16:21:15.221�[0m Nov 23 16:21:15.288: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.com2bglw" �[1mSTEP:�[0m Deleting sc �[38;5;243m11/23/22 16:21:15.357�[0m [AfterEach] [Testpattern: Dynamic PV (block volmode)] volumes test/e2e/framework/framework.go:187 �[1mSTEP:�[0m Collecting events from namespace "volume-9370". �[38;5;243m11/23/22 16:21:15.425�[0m �[1mSTEP:�[0m Found 5 events. �[38;5;243m11/23/22 16:21:15.492�[0m Nov 23 16:21:15.493: INFO: At 2022-11-23 16:16:14 +0000 UTC - event for ebs.csi.aws.com2bglw: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 16:21:15.493: INFO: At 2022-11-23 16:16:14 +0000 UTC - event for ebs.csi.aws.com2bglw: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 16:21:15.493: INFO: At 2022-11-23 16:16:14 +0000 UTC - event for ebs.csi.aws.com2bglw: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } Provisioning: External provisioner is provisioning volume for claim "volume-9370/ebs.csi.aws.com2bglw" Nov 23 16:21:15.493: INFO: At 2022-11-23 16:16:24 +0000 UTC - event for ebs.csi.aws.com2bglw: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "volume-9370-e2e-scktnsj": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 16:21:15.493: INFO: At 2022-11-23 16:17:01 +0000 UTC - event for ebs.csi.aws.com2bglw: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "volume-9370-e2e-scktnsj": rpc error: code = Internal desc = Could not create volume "pvc-59fab33f-ec4c-4252-9c5a-19d3f451ff87": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Nov 23 16:21:15.559: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 16:21:15.559: INFO: Nov 23 16:21:15.628: INFO: Logging node info for node i-060d86c6ffc5622b0 Nov 23 16:21:15.695: INFO: Node Info: &Node{ObjectMeta:{i-060d86c6ffc5622b0 b49ff0b3-8686-4c0f-b991-1d427ca43e2a 27298 0 2022-11-23 16:02:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-060d86c6ffc5622b0 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.40.57 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-060d86c6ffc5622b0"} io.cilium.network.ipv4-cilium-host:172.20.57.204 io.cilium.network.ipv4-health-ip:172.20.37.200 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 16:02:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2022-11-23 16:03:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:03:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-060d86c6ffc5622b0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:04:42 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:03:24 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.57,},NodeAddress{Type:ExternalIP,Address:54.200.161.23,},NodeAddress{Type:InternalDNS,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-200-161-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec226e0d77c347467223a292b33eb0c5,SystemUUID:ec226e0d-77c3-4746-7223-a292b33eb0c5,BootID:936b9dd2-17e7-49ac-8c1e-7556bf2c9ee8,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.25.4],SizeBytes:129085904,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.25.4],SizeBytes:118436153,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.25.4],SizeBytes:51921208,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42928334,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41477333,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:85d3f1e9dacc72531445989bb10999e1e70ebc409d11be57e5baa5f031a893b0 registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1],SizeBytes:18257577,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955428,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:21:15.695: INFO: Logging kubelet events for node i-060d86c6ffc5622b0 Nov 23 16:21:15.765: INFO: Logging pods the kubelet thinks is on node i-060d86c6ffc5622b0 Nov 23 16:21:15.838: INFO: cilium-operator-7f7bcdf787-xnz2q started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:15.838: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 16:21:15.838: INFO: aws-cloud-controller-manager-dzzkp started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:15.838: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Nov 23 16:21:15.838: INFO: etcd-manager-main-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:15.838: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 16:21:15.838: INFO: kube-apiserver-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+2 container statuses recorded) Nov 23 16:21:15.838: INFO: Container healthcheck ready: true, restart count 0 Nov 23 16:21:15.838: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 16:21:15.838: INFO: kube-controller-manager-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:15.838: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 16:21:15.838: INFO: kube-scheduler-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:15.838: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 16:21:15.838: INFO: dns-controller-89c6d5f67-9hm78 started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:15.838: INFO: Container dns-controller ready: true, restart count 0 Nov 23 16:21:15.838: INFO: cilium-f2gg7 started at 2022-11-23 16:03:06 +0000 UTC (1+1 container statuses recorded) Nov 23 16:21:15.838: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:21:15.838: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 16:21:15.838: INFO: ebs-csi-node-vpv4t started at 2022-11-23 16:03:06 +0000 UTC (0+3 container statuses recorded) Nov 23 16:21:15.838: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:21:15.838: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:21:15.838: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:21:15.838: INFO: etcd-manager-events-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:15.838: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 16:21:15.838: INFO: kops-controller-7w98n started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:15.838: INFO: Container kops-controller ready: true, restart count 0 Nov 23 16:21:16.108: INFO: Latency metrics for node i-060d86c6ffc5622b0 Nov 23 16:21:16.108: INFO: Logging node info for node i-0765e732a9635923b Nov 23 16:21:16.175: INFO: Node Info: &Node{ObjectMeta:{i-0765e732a9635923b 10981661-cc27-412a-9e66-7a37930b898f 27701 0 2022-11-23 16:04:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0765e732a9635923b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0765e732a9635923b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.58.74 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0765e732a9635923b"} io.cilium.network.ipv4-cilium-host:172.20.54.198 io.cilium.network.ipv4-health-ip:172.20.60.24 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:07 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:20:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0765e732a9635923b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:07 +0000 UTC,LastTransitionTime:2022-11-23 16:05:07 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:47 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:47 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:47 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:20:47 +0000 UTC,LastTransitionTime:2022-11-23 16:05:00 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.58.74,},NodeAddress{Type:ExternalIP,Address:52.25.125.23,},NodeAddress{Type:InternalDNS,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-25-125-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec23d129b4036d8d979053444dfae008,SystemUUID:ec23d129-b403-6d8d-9790-53444dfae008,BootID:fe390540-4601-456e-9686-93a36ea3dc8f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:21:16.175: INFO: Logging kubelet events for node i-0765e732a9635923b Nov 23 16:21:16.246: INFO: Logging pods the kubelet thinks is on node i-0765e732a9635923b Nov 23 16:21:16.318: INFO: suspend-false-to-true-wrh9j started at 2022-11-23 16:20:42 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:16.318: INFO: Container c ready: true, restart count 0 Nov 23 16:21:16.318: INFO: oidc-discovery-validator started at 2022-11-23 16:19:58 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:16.318: INFO: Container oidc-discovery-validator ready: false, restart count 0 Nov 23 16:21:16.318: INFO: suspend-false-to-true-8vhfh started at 2022-11-23 16:20:42 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:16.318: INFO: Container c ready: true, restart count 0 Nov 23 16:21:16.318: INFO: cilium-czwdq started at 2022-11-23 16:04:40 +0000 UTC (1+1 container statuses recorded) Nov 23 16:21:16.318: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:21:16.318: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:21:16.318: INFO: deployment-97c6466db-trmfq started at 2022-11-23 16:20:52 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:16.318: INFO: Container nginx ready: false, restart count 0 Nov 23 16:21:16.318: INFO: coredns-6c7bddbb75-2n2dw started at 2022-11-23 16:05:19 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:16.318: INFO: Container coredns ready: true, restart count 0 Nov 23 16:21:16.318: INFO: test-rolling-update-with-lb-dbbf5f9b8-hp5hf started at 2022-11-23 16:21:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:16.318: INFO: Container agnhost ready: true, restart count 0 Nov 23 16:21:16.318: INFO: ebs-csi-node-5khlc started at 2022-11-23 16:04:40 +0000 UTC (0+3 container statuses recorded) Nov 23 16:21:16.318: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:21:16.318: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:21:16.318: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:21:16.600: INFO: Latency metrics for node i-0765e732a9635923b Nov 23 16:21:16.600: INFO: Logging node info for node i-0ade3c3082581cb52 Nov 23 16:21:16.679: INFO: Node Info: &Node{ObjectMeta:{i-0ade3c3082581cb52 2305662a-7d05-4979-bcec-220b8b4672db 28230 0 2022-11-23 16:04:42 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0ade3c3082581cb52 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0ade3c3082581cb52 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.56.12 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-3364":"i-0ade3c3082581cb52","ebs.csi.aws.com":"i-0ade3c3082581cb52"} io.cilium.network.ipv4-cilium-host:172.20.45.237 io.cilium.network.ipv4-health-ip:172.20.55.90 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:21:06 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 16:21:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0ade3c3082581cb52,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:12 +0000 UTC,LastTransitionTime:2022-11-23 16:05:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:11 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:11 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:11 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:21:11 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.56.12,},NodeAddress{Type:ExternalIP,Address:35.92.243.169,},NodeAddress{Type:InternalDNS,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-92-243-169.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bd980dfb52643cb4857d57b45824b,SystemUUID:ec2bd980-dfb5-2643-cb48-57d57b45824b,BootID:55912341-dce8-4199-a57c-c8703e24af46,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 registry.k8s.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-3364^d43a7297-6b4a-11ed-801d-6e2eb92aaf3f],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-3364^d43a7297-6b4a-11ed-801d-6e2eb92aaf3f,DevicePath:,},},Config:nil,},} Nov 23 16:21:16.679: INFO: Logging kubelet events for node i-0ade3c3082581cb52 Nov 23 16:21:16.750: INFO: Logging pods the kubelet thinks is on node i-0ade3c3082581cb52 Nov 23 16:21:16.834: INFO: csi-hostpathplugin-0 started at 2022-11-23 16:21:01 +0000 UTC (0+7 container statuses recorded) Nov 23 16:21:16.834: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:21:16.834: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:21:16.834: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:21:16.834: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 23 16:21:16.834: INFO: Container hostpath ready: true, restart count 0 Nov 23 16:21:16.834: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:21:16.834: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:21:16.834: INFO: hostexec-i-0ade3c3082581cb52-h72bd started at 2022-11-23 16:20:37 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:16.834: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 16:21:16.834: INFO: test-rolling-update-with-lb-dbbf5f9b8-bhfzj started at 2022-11-23 16:21:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:16.834: INFO: Container agnhost ready: true, restart count 0 Nov 23 16:21:16.834: INFO: cilium-q5tsk started at 2022-11-23 16:04:43 +0000 UTC (1+1 container statuses recorded) Nov 23 16:21:16.834: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:21:16.834: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:21:16.834: INFO: pod-subpath-test-dynamicpv-tl29 started at 2022-11-23 16:21:06 +0000 UTC (2+1 container statuses recorded) Nov 23 16:21:16.834: INFO: Init container init-volume-dynamicpv-tl29 ready: true, restart count 0 Nov 23 16:21:16.834: INFO: Init container test-init-volume-dynamicpv-tl29 ready: true, restart count 0 Nov 23 16:21:16.834: INFO: Container test-container-subpath-dynamicpv-tl29 ready: false, restart count 0 Nov 23 16:21:16.834: INFO: ebs-csi-node-t56sd started at 2022-11-23 16:04:43 +0000 UTC (0+3 container statuses recorded) Nov 23 16:21:16.834: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:21:16.834: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:21:16.834: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:21:16.834: INFO: pod-subpath-test-preprovisionedpv-t22w started at 2022-11-23 16:20:51 +0000 UTC (1+1 container statuses recorded) Nov 23 16:21:16.834: INFO: Init container init-volume-preprovisionedpv-t22w ready: true, restart count 0 Nov 23 16:21:16.834: INFO: Container test-container-subpath-preprovisionedpv-t22w ready: false, restart count 0 Nov 23 16:21:17.172: INFO: Latency metrics for node i-0ade3c3082581cb52 Nov 23 16:21:17.172: INFO: Logging node info for node i-0bf2756d84bae9168 Nov 23 16:21:17.238: INFO: Node Info: &Node{ObjectMeta:{i-0bf2756d84bae9168 2f26f127-5eb0-44f3-97f0-990045e7b900 27933 0 2022-11-23 16:04:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0bf2756d84bae9168 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0bf2756d84bae9168 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.218 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0bf2756d84bae9168"} io.cilium.network.ipv4-cilium-host:172.20.35.201 io.cilium.network.ipv4-health-ip:172.20.51.170 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:02 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:20:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0bf2756d84bae9168,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:02 +0000 UTC,LastTransitionTime:2022-11-23 16:05:02 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.218,},NodeAddress{Type:ExternalIP,Address:34.221.37.130,},NodeAddress{Type:InternalDNS,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-37-130.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d7db7f0ff759b7a6209e832f1dcea,SystemUUID:ec2d7db7-f0ff-759b-7a62-09e832f1dcea,BootID:8f294a85-a1ed-4dfb-bdb9-f8818e1f067f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f registry.k8s.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:21:17.239: INFO: Logging kubelet events for node i-0bf2756d84bae9168 Nov 23 16:21:17.309: INFO: Logging pods the kubelet thinks is on node i-0bf2756d84bae9168 Nov 23 16:21:17.380: INFO: cilium-scm24 started at 2022-11-23 16:04:35 +0000 UTC (1+1 container statuses recorded) Nov 23 16:21:17.380: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:21:17.380: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:21:17.380: INFO: test-rolling-update-with-lb-dbbf5f9b8-7fpt7 started at 2022-11-23 16:21:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:17.380: INFO: Container agnhost ready: true, restart count 0 Nov 23 16:21:17.380: INFO: ebs-csi-node-crshz started at 2022-11-23 16:04:35 +0000 UTC (0+3 container statuses recorded) Nov 23 16:21:17.380: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:21:17.380: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:21:17.380: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:21:17.380: INFO: ebs-csi-controller-847c4b8bc4-kwpsb started at 2022-11-23 16:04:56 +0000 UTC (0+5 container statuses recorded) Nov 23 16:21:17.380: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:21:17.380: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:21:17.380: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:21:17.380: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:21:17.380: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:21:17.656: INFO: Latency metrics for node i-0bf2756d84bae9168 Nov 23 16:21:17.656: INFO: Logging node info for node i-0fe7a5031fb82cc5b Nov 23 16:21:17.724: INFO: Node Info: &Node{ObjectMeta:{i-0fe7a5031fb82cc5b 0297efe3-5070-45f1-8aa2-9a1056b17b59 27351 0 2022-11-23 16:04:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0fe7a5031fb82cc5b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0fe7a5031fb82cc5b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.89 csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-5521":"i-0fe7a5031fb82cc5b","ebs.csi.aws.com":"i-0fe7a5031fb82cc5b"} io.cilium.network.ipv4-cilium-host:172.20.57.57 io.cilium.network.ipv4-health-ip:172.20.59.99 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-11-23 16:04:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:03 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:20:13 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 16:20:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0fe7a5031fb82cc5b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:03 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:13 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:13 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:13 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:20:13 +0000 UTC,LastTransitionTime:2022-11-23 16:04:54 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.89,},NodeAddress{Type:ExternalIP,Address:34.221.99.167,},NodeAddress{Type:InternalDNS,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-99-167.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29292cb2c053c780af3b39ea2fa8c6,SystemUUID:ec29292c-b2c0-53c7-80af-3b39ea2fa8c6,BootID:ff4df73e-ccb2-4109-99ca-04a1b7ab5611,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-5521^90b3fba0-6b4a-11ed-be42-664930db81b2],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-5521^90b3fba0-6b4a-11ed-be42-664930db81b2,DevicePath:,},},Config:nil,},} Nov 23 16:21:17.724: INFO: Logging kubelet events for node i-0fe7a5031fb82cc5b Nov 23 16:21:17.800: INFO: Logging pods the kubelet thinks is on node i-0fe7a5031fb82cc5b Nov 23 16:21:17.872: INFO: cilium-727pv started at 2022-11-23 16:04:34 +0000 UTC (1+1 container statuses recorded) Nov 23 16:21:17.872: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:21:17.872: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:21:17.872: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 16:19:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:17.872: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:21:17.872: INFO: coredns-6c7bddbb75-bpdc6 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:17.872: INFO: Container coredns ready: true, restart count 0 Nov 23 16:21:17.872: INFO: coredns-autoscaler-db7b97744-r4jm7 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:17.872: INFO: Container autoscaler ready: true, restart count 0 Nov 23 16:21:17.872: INFO: hostexec-i-0fe7a5031fb82cc5b-lllwr started at 2022-11-23 16:21:05 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:17.872: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 16:21:17.872: INFO: ebs-csi-controller-847c4b8bc4-lxdkr started at 2022-11-23 16:04:54 +0000 UTC (0+5 container statuses recorded) Nov 23 16:21:17.872: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:21:17.872: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:21:17.872: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:21:17.872: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:21:17.872: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:21:17.872: INFO: csi-mockplugin-0 started at 2022-11-23 16:19:06 +0000 UTC (0+3 container statuses recorded) Nov 23 16:21:17.872: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:21:17.872: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 16:21:17.872: INFO: Container mock ready: true, restart count 0 Nov 23 16:21:17.872: INFO: pvc-volume-tester-l5df6 started at 2022-11-23 16:19:12 +0000 UTC (0+1 container statuses recorded) Nov 23 16:21:17.872: INFO: Container volume-tester ready: true, restart count 0 Nov 23 16:21:17.872: INFO: ebs-csi-node-8ffwl started at 2022-11-23 16:04:34 +0000 UTC (0+3 container statuses recorded) Nov 23 16:21:17.872: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:21:17.872: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:21:17.872: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:21:18.178: INFO: Latency metrics for node i-0fe7a5031fb82cc5b Nov 23 16:21:18.178: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP:�[0m Destroying namespace "volume-9370" for this suite. �[38;5;243m11/23/22 16:21:18.246�[0m
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\(allowExpansion\)\]\svolume\-expand\sVerify\sif\soffline\sPVC\sexpansion\sworks$'
test/e2e/storage/testsuites/volume_expand.go:198 k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func4() test/e2e/storage/testsuites/volume_expand.go:198 +0x3bafrom junit_01.xml
{"msg":"FAILED External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works","completed":2,"skipped":28,"failed":1,"failures":["External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works"]} [BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand test/e2e/framework/framework.go:186 �[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m11/23/22 16:08:04.627�[0m Nov 23 16:08:04.627: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP:�[0m Building a namespace api object, basename volume-expand �[38;5;243m11/23/22 16:08:04.628�[0m �[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m11/23/22 16:08:04.829�[0m �[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m11/23/22 16:08:04.961�[0m [It] Verify if offline PVC expansion works test/e2e/storage/testsuites/volume_expand.go:176 Nov 23 16:08:05.097: INFO: Creating resource for dynamic PV Nov 23 16:08:05.097: INFO: Using claimSize:1Gi, test suite supported size:{ 1Gi}, driver(ebs.csi.aws.com) supported size:{ 1Gi} �[1mSTEP:�[0m creating a StorageClass volume-expand-5378-e2e-scngbck �[38;5;243m11/23/22 16:08:05.098�[0m �[1mSTEP:�[0m creating a claim �[38;5;243m11/23/22 16:08:05.185�[0m Nov 23 16:08:05.185: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP:�[0m Creating a pod with dynamically provisioned volume �[38;5;243m11/23/22 16:08:05.384�[0m Nov 23 16:08:05.461: INFO: Waiting up to 5m0s for pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d" in namespace "volume-expand-5378" to be "running" Nov 23 16:08:05.528: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 66.759774ms Nov 23 16:08:07.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2.133549878s Nov 23 16:08:09.597: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4.135488524s Nov 23 16:08:11.597: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 6.136050022s Nov 23 16:08:13.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 8.134255401s Nov 23 16:08:15.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 10.133989104s Nov 23 16:08:17.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 12.134209815s Nov 23 16:08:19.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 14.13358317s Nov 23 16:08:21.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 16.135007225s Nov 23 16:08:23.602: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 18.140749514s Nov 23 16:08:25.598: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 20.136743822s Nov 23 16:08:27.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 22.134108296s Nov 23 16:08:29.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 24.133964543s Nov 23 16:08:31.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 26.1338093s Nov 23 16:08:33.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 28.134601121s Nov 23 16:08:35.597: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 30.135136446s Nov 23 16:08:37.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 32.134815022s Nov 23 16:08:39.609: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 34.147860595s Nov 23 16:08:41.705: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 36.243228263s Nov 23 16:08:43.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 38.134935547s Nov 23 16:08:45.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 40.133682362s Nov 23 16:08:47.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 42.134081689s Nov 23 16:08:49.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 44.133543158s Nov 23 16:08:51.597: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 46.135454151s Nov 23 16:08:53.599: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 48.137235158s Nov 23 16:08:55.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 50.134163471s Nov 23 16:08:57.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 52.134583212s Nov 23 16:08:59.597: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 54.135541962s Nov 23 16:09:01.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 56.13402354s Nov 23 16:09:03.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 58.134393396s Nov 23 16:09:05.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.134189117s Nov 23 16:09:07.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.133710183s Nov 23 16:09:09.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.133521109s Nov 23 16:09:11.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.134328954s Nov 23 16:09:13.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.133573703s Nov 23 16:09:15.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.134786032s Nov 23 16:09:17.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.13446489s Nov 23 16:09:19.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.133896724s Nov 23 16:09:21.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.13497835s Nov 23 16:09:23.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.134235929s Nov 23 16:09:25.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.134137875s Nov 23 16:09:27.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.134284386s Nov 23 16:09:29.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.133975969s Nov 23 16:09:31.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.133461082s Nov 23 16:09:33.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.13445642s Nov 23 16:09:35.597: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.136059048s Nov 23 16:09:37.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.133701916s Nov 23 16:09:39.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.133736855s Nov 23 16:09:41.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.134469508s Nov 23 16:09:43.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m38.134280557s Nov 23 16:09:45.597: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m40.135266213s Nov 23 16:09:47.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m42.134099294s Nov 23 16:09:49.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.133655017s Nov 23 16:09:51.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.134526969s Nov 23 16:09:53.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.1339993s Nov 23 16:09:55.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m50.134449658s Nov 23 16:09:57.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m52.133576599s Nov 23 16:09:59.597: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m54.135372965s Nov 23 16:10:01.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m56.134431877s Nov 23 16:10:03.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m58.134385251s Nov 23 16:10:05.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m0.133570359s Nov 23 16:10:07.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m2.134902897s Nov 23 16:10:09.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m4.134199197s Nov 23 16:10:11.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.134107531s Nov 23 16:10:13.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.133963437s Nov 23 16:10:15.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.133984114s Nov 23 16:10:17.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.133859527s Nov 23 16:10:19.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.134341077s Nov 23 16:10:21.603: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.142052488s Nov 23 16:10:23.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.133300759s Nov 23 16:10:25.603: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.141592296s Nov 23 16:10:27.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.133360925s Nov 23 16:10:29.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.133699135s Nov 23 16:10:31.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.133497789s Nov 23 16:10:33.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.133866498s Nov 23 16:10:35.603: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.141190203s Nov 23 16:10:37.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.133594104s Nov 23 16:10:39.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.134919282s Nov 23 16:10:41.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.134265286s Nov 23 16:10:43.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.134473019s Nov 23 16:10:45.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.133706739s Nov 23 16:10:47.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.133953041s Nov 23 16:10:49.599: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.13771639s Nov 23 16:10:51.601: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.139180494s Nov 23 16:10:53.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.134899535s Nov 23 16:10:55.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.133697947s Nov 23 16:10:57.599: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.137861216s Nov 23 16:10:59.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.133799964s Nov 23 16:11:01.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.134425027s Nov 23 16:11:03.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 2m58.133681375s Nov 23 16:11:05.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m0.133898239s Nov 23 16:11:07.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m2.134011868s Nov 23 16:11:09.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.133708528s Nov 23 16:11:11.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m6.133629441s Nov 23 16:11:13.599: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m8.137500664s Nov 23 16:11:15.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.133231671s Nov 23 16:11:17.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.133401678s Nov 23 16:11:19.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m14.133537855s Nov 23 16:11:21.618: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m16.156894008s Nov 23 16:11:23.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m18.134640582s Nov 23 16:11:25.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m20.133467259s Nov 23 16:11:27.598: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m22.136983132s Nov 23 16:11:29.594: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.133065498s Nov 23 16:11:31.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.133735599s Nov 23 16:11:33.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.133596648s Nov 23 16:11:35.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.133349821s Nov 23 16:11:37.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.133835086s Nov 23 16:11:39.597: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m34.135235214s Nov 23 16:11:41.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m36.133638465s Nov 23 16:11:43.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m38.133596206s Nov 23 16:11:45.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m40.134553845s Nov 23 16:11:47.597: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m42.135327607s Nov 23 16:11:49.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m44.134606949s Nov 23 16:11:51.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m46.133899834s Nov 23 16:11:53.597: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m48.135763461s Nov 23 16:11:55.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m50.134721638s Nov 23 16:11:57.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m52.133827169s Nov 23 16:11:59.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m54.133901399s Nov 23 16:12:01.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m56.133675328s Nov 23 16:12:03.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 3m58.134413813s Nov 23 16:12:05.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m0.134227188s Nov 23 16:12:07.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m2.134083838s Nov 23 16:12:09.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m4.13471706s Nov 23 16:12:11.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.133472122s Nov 23 16:12:13.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.134112291s Nov 23 16:12:15.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.134967668s Nov 23 16:12:17.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.133840056s Nov 23 16:12:19.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.133866309s Nov 23 16:12:21.599: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m16.137986058s Nov 23 16:12:23.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m18.133726303s Nov 23 16:12:25.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m20.134267363s Nov 23 16:12:27.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m22.134495915s Nov 23 16:12:29.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m24.1335835s Nov 23 16:12:31.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.133695585s Nov 23 16:12:33.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.133430186s Nov 23 16:12:35.597: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.135420365s Nov 23 16:12:37.605: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.143155742s Nov 23 16:12:39.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.134751849s Nov 23 16:12:41.597: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.135434085s Nov 23 16:12:43.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.133687693s Nov 23 16:12:45.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.133837367s Nov 23 16:12:47.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.135063741s Nov 23 16:12:49.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.133598478s Nov 23 16:12:51.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.134468848s Nov 23 16:12:53.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.133576553s Nov 23 16:12:55.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.134214577s Nov 23 16:12:57.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.133902082s Nov 23 16:12:59.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.133855865s Nov 23 16:13:01.595: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.134048368s Nov 23 16:13:03.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.134092466s Nov 23 16:13:05.596: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.134325731s Nov 23 16:13:05.662: INFO: Pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.200506771s Nov 23 16:13:05.662: INFO: Unexpected error: While creating pods for resizing: <*errors.errorString | 0xc000c372c0>: { s: "pod \"pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d\" is not Running: timed out while waiting for pod volume-expand-5378/pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d to be running", } Nov 23 16:13:05.662: FAIL: While creating pods for resizing: pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d" is not Running: timed out while waiting for pod volume-expand-5378/pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d to be running Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func4() test/e2e/storage/testsuites/volume_expand.go:198 +0x3ba Nov 23 16:13:05.662: INFO: Deleting pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d" in namespace "volume-expand-5378" Nov 23 16:13:05.735: INFO: Wait up to 5m0s for pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d" to be fully deleted �[1mSTEP:�[0m Deleting pod �[38;5;243m11/23/22 16:13:05.801�[0m Nov 23 16:13:05.801: INFO: Deleting pod "pod-ebfcd9e5-b990-4a44-b46c-2f9b2749893d" in namespace "volume-expand-5378" �[1mSTEP:�[0m Deleting pvc �[38;5;243m11/23/22 16:13:05.867�[0m Nov 23 16:13:05.934: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comdd2bh" �[1mSTEP:�[0m Deleting sc �[38;5;243m11/23/22 16:13:06.004�[0m [AfterEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand test/e2e/framework/framework.go:187 �[1mSTEP:�[0m Collecting events from namespace "volume-expand-5378". �[38;5;243m11/23/22 16:13:06.072�[0m �[1mSTEP:�[0m Found 5 events. �[38;5;243m11/23/22 16:13:06.139�[0m Nov 23 16:13:06.139: INFO: At 2022-11-23 16:08:05 +0000 UTC - event for ebs.csi.aws.comdd2bh: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 16:13:06.139: INFO: At 2022-11-23 16:08:05 +0000 UTC - event for ebs.csi.aws.comdd2bh: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 16:13:06.139: INFO: At 2022-11-23 16:08:05 +0000 UTC - event for ebs.csi.aws.comdd2bh: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-5378/ebs.csi.aws.comdd2bh" Nov 23 16:13:06.139: INFO: At 2022-11-23 16:08:15 +0000 UTC - event for ebs.csi.aws.comdd2bh: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-5378-e2e-scngbck": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 16:13:06.139: INFO: At 2022-11-23 16:08:52 +0000 UTC - event for ebs.csi.aws.comdd2bh: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-5378-e2e-scngbck": rpc error: code = Internal desc = Could not create volume "pvc-aa2d2ae6-d707-4a49-937a-0816f1bb282d": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Nov 23 16:13:06.205: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 16:13:06.205: INFO: Nov 23 16:13:06.274: INFO: Logging node info for node i-060d86c6ffc5622b0 Nov 23 16:13:06.340: INFO: Node Info: &Node{ObjectMeta:{i-060d86c6ffc5622b0 b49ff0b3-8686-4c0f-b991-1d427ca43e2a 7999 0 2022-11-23 16:02:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-060d86c6ffc5622b0 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.40.57 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-060d86c6ffc5622b0"} io.cilium.network.ipv4-cilium-host:172.20.57.204 io.cilium.network.ipv4-health-ip:172.20.37.200 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 16:02:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2022-11-23 16:03:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:03:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:10:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-060d86c6ffc5622b0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:04:42 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:10:12 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:10:12 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:10:12 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:10:12 +0000 UTC,LastTransitionTime:2022-11-23 16:03:24 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.57,},NodeAddress{Type:ExternalIP,Address:54.200.161.23,},NodeAddress{Type:InternalDNS,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-200-161-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec226e0d77c347467223a292b33eb0c5,SystemUUID:ec226e0d-77c3-4746-7223-a292b33eb0c5,BootID:936b9dd2-17e7-49ac-8c1e-7556bf2c9ee8,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.25.4],SizeBytes:129085904,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.25.4],SizeBytes:118436153,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.25.4],SizeBytes:51921208,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42928334,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41477333,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:85d3f1e9dacc72531445989bb10999e1e70ebc409d11be57e5baa5f031a893b0 registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1],SizeBytes:18257577,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955428,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:13:06.341: INFO: Logging kubelet events for node i-060d86c6ffc5622b0 Nov 23 16:13:06.411: INFO: Logging pods the kubelet thinks is on node i-060d86c6ffc5622b0 Nov 23 16:13:06.556: INFO: etcd-manager-events-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:06.556: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 16:13:06.556: INFO: kops-controller-7w98n started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:06.556: INFO: Container kops-controller ready: true, restart count 0 Nov 23 16:13:06.556: INFO: aws-cloud-controller-manager-dzzkp started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:06.556: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Nov 23 16:13:06.556: INFO: etcd-manager-main-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:06.556: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 16:13:06.556: INFO: kube-apiserver-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+2 container statuses recorded) Nov 23 16:13:06.556: INFO: Container healthcheck ready: true, restart count 0 Nov 23 16:13:06.556: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 16:13:06.556: INFO: kube-controller-manager-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:06.556: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 16:13:06.556: INFO: kube-scheduler-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:06.556: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 16:13:06.556: INFO: dns-controller-89c6d5f67-9hm78 started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:06.556: INFO: Container dns-controller ready: true, restart count 0 Nov 23 16:13:06.556: INFO: cilium-f2gg7 started at 2022-11-23 16:03:06 +0000 UTC (1+1 container statuses recorded) Nov 23 16:13:06.556: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:13:06.556: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 16:13:06.556: INFO: cilium-operator-7f7bcdf787-xnz2q started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:06.556: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 16:13:06.556: INFO: ebs-csi-node-vpv4t started at 2022-11-23 16:03:06 +0000 UTC (0+3 container statuses recorded) Nov 23 16:13:06.556: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:13:06.556: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:13:06.556: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:13:06.804: INFO: Latency metrics for node i-060d86c6ffc5622b0 Nov 23 16:13:06.804: INFO: Logging node info for node i-0765e732a9635923b Nov 23 16:13:06.871: INFO: Node Info: &Node{ObjectMeta:{i-0765e732a9635923b 10981661-cc27-412a-9e66-7a37930b898f 14279 0 2022-11-23 16:04:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0765e732a9635923b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0765e732a9635923b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.58.74 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-5280":"i-0765e732a9635923b","csi-mock-csi-mock-volumes-2288":"csi-mock-csi-mock-volumes-2288","ebs.csi.aws.com":"i-0765e732a9635923b"} io.cilium.network.ipv4-cilium-host:172.20.54.198 io.cilium.network.ipv4-health-ip:172.20.60.24 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:07 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:12:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0765e732a9635923b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:07 +0000 UTC,LastTransitionTime:2022-11-23 16:05:07 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:11:38 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:11:38 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:11:38 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:11:38 +0000 UTC,LastTransitionTime:2022-11-23 16:05:00 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.58.74,},NodeAddress{Type:ExternalIP,Address:52.25.125.23,},NodeAddress{Type:InternalDNS,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-25-125-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec23d129b4036d8d979053444dfae008,SystemUUID:ec23d129-b403-6d8d-9790-53444dfae008,BootID:fe390540-4601-456e-9686-93a36ea3dc8f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:13:06.872: INFO: Logging kubelet events for node i-0765e732a9635923b Nov 23 16:13:06.950: INFO: Logging pods the kubelet thinks is on node i-0765e732a9635923b Nov 23 16:13:07.022: INFO: csi-mockplugin-0 started at 2022-11-23 16:12:39 +0000 UTC (0+4 container statuses recorded) Nov 23 16:13:07.022: INFO: Container busybox ready: true, restart count 0 Nov 23 16:13:07.022: INFO: Container csi-provisioner ready: false, restart count 1 Nov 23 16:13:07.022: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 16:13:07.022: INFO: Container mock ready: true, restart count 0 Nov 23 16:13:07.022: INFO: cilium-czwdq started at 2022-11-23 16:04:40 +0000 UTC (1+1 container statuses recorded) Nov 23 16:13:07.022: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:13:07.022: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:13:07.022: INFO: coredns-6c7bddbb75-2n2dw started at 2022-11-23 16:05:19 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:07.022: INFO: Container coredns ready: true, restart count 0 Nov 23 16:13:07.022: INFO: ebs-csi-node-5khlc started at 2022-11-23 16:04:40 +0000 UTC (0+3 container statuses recorded) Nov 23 16:13:07.022: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:13:07.022: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:13:07.022: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:13:07.022: INFO: netserver-0 started at 2022-11-23 16:12:47 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:07.022: INFO: Container webserver ready: false, restart count 0 Nov 23 16:13:07.022: INFO: csi-hostpathplugin-0 started at 2022-11-23 16:12:20 +0000 UTC (0+7 container statuses recorded) Nov 23 16:13:07.022: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:13:07.022: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:13:07.022: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:13:07.022: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 23 16:13:07.022: INFO: Container hostpath ready: true, restart count 0 Nov 23 16:13:07.022: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:13:07.022: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:13:07.370: INFO: Latency metrics for node i-0765e732a9635923b Nov 23 16:13:07.370: INFO: Logging node info for node i-0ade3c3082581cb52 Nov 23 16:13:07.438: INFO: Node Info: &Node{ObjectMeta:{i-0ade3c3082581cb52 2305662a-7d05-4979-bcec-220b8b4672db 13497 0 2022-11-23 16:04:42 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0ade3c3082581cb52 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0ade3c3082581cb52 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.56.12 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0ade3c3082581cb52"} io.cilium.network.ipv4-cilium-host:172.20.45.237 io.cilium.network.ipv4-health-ip:172.20.55.90 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:12:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0ade3c3082581cb52,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:12 +0000 UTC,LastTransitionTime:2022-11-23 16:05:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:12:21 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:12:21 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:12:21 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:12:21 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.56.12,},NodeAddress{Type:ExternalIP,Address:35.92.243.169,},NodeAddress{Type:InternalDNS,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-92-243-169.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bd980dfb52643cb4857d57b45824b,SystemUUID:ec2bd980-dfb5-2643-cb48-57d57b45824b,BootID:55912341-dce8-4199-a57c-c8703e24af46,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:13:07.439: INFO: Logging kubelet events for node i-0ade3c3082581cb52 Nov 23 16:13:07.508: INFO: Logging pods the kubelet thinks is on node i-0ade3c3082581cb52 Nov 23 16:13:07.581: INFO: hostexec-i-0ade3c3082581cb52-9nhjk started at 2022-11-23 16:13:04 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:07.581: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 16:13:07.581: INFO: cilium-q5tsk started at 2022-11-23 16:04:43 +0000 UTC (1+1 container statuses recorded) Nov 23 16:13:07.581: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:13:07.581: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:13:07.581: INFO: liveness-38645321-d2f1-4818-b657-547947353222 started at 2022-11-23 16:12:51 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:07.581: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 16:13:07.581: INFO: pod-1 started at 2022-11-23 16:13:05 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:07.581: INFO: Container donothing ready: false, restart count 0 Nov 23 16:13:07.581: INFO: pod-2 started at 2022-11-23 16:13:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:07.581: INFO: Container donothing ready: false, restart count 0 Nov 23 16:13:07.581: INFO: netserver-1 started at 2022-11-23 16:12:47 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:07.581: INFO: Container webserver ready: false, restart count 0 Nov 23 16:13:07.581: INFO: pod-0 started at 2022-11-23 16:13:05 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:07.581: INFO: Container donothing ready: false, restart count 0 Nov 23 16:13:07.581: INFO: ebs-csi-node-t56sd started at 2022-11-23 16:04:43 +0000 UTC (0+3 container statuses recorded) Nov 23 16:13:07.581: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:13:07.581: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:13:07.581: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:13:07.892: INFO: Latency metrics for node i-0ade3c3082581cb52 Nov 23 16:13:07.892: INFO: Logging node info for node i-0bf2756d84bae9168 Nov 23 16:13:07.958: INFO: Node Info: &Node{ObjectMeta:{i-0bf2756d84bae9168 2f26f127-5eb0-44f3-97f0-990045e7b900 14580 0 2022-11-23 16:04:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0bf2756d84bae9168 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0bf2756d84bae9168 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.218 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-323":"i-0bf2756d84bae9168","csi-hostpath-ephemeral-4263":"i-0bf2756d84bae9168","ebs.csi.aws.com":"i-0bf2756d84bae9168"} io.cilium.network.ipv4-cilium-host:172.20.35.201 io.cilium.network.ipv4-health-ip:172.20.51.170 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:02 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:12:50 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 16:12:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0bf2756d84bae9168,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:02 +0000 UTC,LastTransitionTime:2022-11-23 16:05:02 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:12:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:12:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:12:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:12:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.218,},NodeAddress{Type:ExternalIP,Address:34.221.37.130,},NodeAddress{Type:InternalDNS,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-37-130.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d7db7f0ff759b7a6209e832f1dcea,SystemUUID:ec2d7db7-f0ff-759b-7a62-09e832f1dcea,BootID:8f294a85-a1ed-4dfb-bdb9-f8818e1f067f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f registry.k8s.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-323^a3a2194b-6b49-11ed-8275-0ef81ef9666a kubernetes.io/csi/csi-hostpath-ephemeral-323^ac6cbeb5-6b49-11ed-8275-0ef81ef9666a kubernetes.io/csi/csi-hostpath-ephemeral-4263^a3a6d07a-6b49-11ed-be66-1607fe83facc kubernetes.io/csi/csi-hostpath-ephemeral-4263^ac6c150a-6b49-11ed-be66-1607fe83facc],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-323^a3a2194b-6b49-11ed-8275-0ef81ef9666a,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-323^ac6cbeb5-6b49-11ed-8275-0ef81ef9666a,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-4263^ac6c150a-6b49-11ed-be66-1607fe83facc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-4263^a3a6d07a-6b49-11ed-be66-1607fe83facc,DevicePath:,},},Config:nil,},} Nov 23 16:13:07.959: INFO: Logging kubelet events for node i-0bf2756d84bae9168 Nov 23 16:13:08.029: INFO: Logging pods the kubelet thinks is on node i-0bf2756d84bae9168 Nov 23 16:13:08.104: INFO: ebs-csi-node-crshz started at 2022-11-23 16:04:35 +0000 UTC (0+3 container statuses recorded) Nov 23 16:13:08.104: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:13:08.104: INFO: inline-volume-tester2-vk4mp started at 2022-11-23 16:12:49 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:08.104: INFO: Container csi-volume-tester ready: true, restart count 0 Nov 23 16:13:08.104: INFO: pod-subpath-test-preprovisionedpv-w75n started at 2022-11-23 16:12:51 +0000 UTC (1+1 container statuses recorded) Nov 23 16:13:08.104: INFO: Init container init-volume-preprovisionedpv-w75n ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container test-container-subpath-preprovisionedpv-w75n ready: true, restart count 0 Nov 23 16:13:08.104: INFO: csi-hostpathplugin-0 started at 2022-11-23 16:12:31 +0000 UTC (0+7 container statuses recorded) Nov 23 16:13:08.104: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container hostpath ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:13:08.104: INFO: netserver-2 started at 2022-11-23 16:12:47 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:08.104: INFO: Container webserver ready: true, restart count 0 Nov 23 16:13:08.104: INFO: cilium-scm24 started at 2022-11-23 16:04:35 +0000 UTC (1+1 container statuses recorded) Nov 23 16:13:08.104: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:13:08.104: INFO: bin-falseea34ec51-25e1-4ca2-9be6-3636e5802eb2 started at 2022-11-23 16:11:55 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:08.104: INFO: Container bin-falseea34ec51-25e1-4ca2-9be6-3636e5802eb2 ready: false, restart count 0 Nov 23 16:13:08.104: INFO: csi-hostpathplugin-0 started at 2022-11-23 16:12:31 +0000 UTC (0+7 container statuses recorded) Nov 23 16:13:08.104: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container hostpath ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:13:08.104: INFO: inline-volume-tester-pwczq started at 2022-11-23 16:12:34 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:08.104: INFO: Container csi-volume-tester ready: true, restart count 0 Nov 23 16:13:08.104: INFO: inline-volume-tester2-mgfgt started at 2022-11-23 16:12:49 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:08.104: INFO: Container csi-volume-tester ready: true, restart count 0 Nov 23 16:13:08.104: INFO: ebs-csi-controller-847c4b8bc4-kwpsb started at 2022-11-23 16:04:56 +0000 UTC (0+5 container statuses recorded) Nov 23 16:13:08.104: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:13:08.104: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:13:08.104: INFO: inline-volume-tester-5svhb started at 2022-11-23 16:12:34 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:08.104: INFO: Container csi-volume-tester ready: true, restart count 0 Nov 23 16:13:08.104: INFO: hostexec-i-0bf2756d84bae9168-6xcjs started at 2022-11-23 16:12:42 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:08.104: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 16:13:08.434: INFO: Latency metrics for node i-0bf2756d84bae9168 Nov 23 16:13:08.434: INFO: Logging node info for node i-0fe7a5031fb82cc5b Nov 23 16:13:08.501: INFO: Node Info: &Node{ObjectMeta:{i-0fe7a5031fb82cc5b 0297efe3-5070-45f1-8aa2-9a1056b17b59 14569 0 2022-11-23 16:04:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0fe7a5031fb82cc5b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0fe7a5031fb82cc5b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.89 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-9022":"i-0fe7a5031fb82cc5b","ebs.csi.aws.com":"i-0fe7a5031fb82cc5b"} io.cilium.network.ipv4-cilium-host:172.20.57.57 io.cilium.network.ipv4-health-ip:172.20.59.99 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-11-23 16:04:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:03 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:12:49 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 16:12:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0fe7a5031fb82cc5b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:03 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:12:54 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:12:54 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:12:54 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:12:54 +0000 UTC,LastTransitionTime:2022-11-23 16:04:54 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.89,},NodeAddress{Type:ExternalIP,Address:34.221.99.167,},NodeAddress{Type:InternalDNS,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-99-167.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29292cb2c053c780af3b39ea2fa8c6,SystemUUID:ec29292c-b2c0-53c7-80af-3b39ea2fa8c6,BootID:ff4df73e-ccb2-4109-99ca-04a1b7ab5611,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-9022^abff0a74-6b49-11ed-a24b-1ad23f1b85fc],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-9022^abff0a74-6b49-11ed-a24b-1ad23f1b85fc,DevicePath:,},},Config:nil,},} Nov 23 16:13:08.501: INFO: Logging kubelet events for node i-0fe7a5031fb82cc5b Nov 23 16:13:08.570: INFO: Logging pods the kubelet thinks is on node i-0fe7a5031fb82cc5b Nov 23 16:13:08.645: INFO: ebs-csi-node-8ffwl started at 2022-11-23 16:04:34 +0000 UTC (0+3 container statuses recorded) Nov 23 16:13:08.645: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:13:08.645: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:13:08.645: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:13:08.645: INFO: coredns-autoscaler-db7b97744-r4jm7 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:08.645: INFO: Container autoscaler ready: true, restart count 0 Nov 23 16:13:08.645: INFO: cilium-727pv started at 2022-11-23 16:04:34 +0000 UTC (1+1 container statuses recorded) Nov 23 16:13:08.645: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:13:08.645: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:13:08.645: INFO: ebs-csi-controller-847c4b8bc4-lxdkr started at 2022-11-23 16:04:54 +0000 UTC (0+5 container statuses recorded) Nov 23 16:13:08.645: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:13:08.645: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:13:08.645: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:13:08.645: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:13:08.645: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:13:08.645: INFO: csi-mockplugin-0 started at 2022-11-23 16:12:18 +0000 UTC (0+4 container statuses recorded) Nov 23 16:13:08.645: INFO: Container busybox ready: true, restart count 0 Nov 23 16:13:08.645: INFO: Container csi-provisioner ready: true, restart count 2 Nov 23 16:13:08.645: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 16:13:08.645: INFO: Container mock ready: true, restart count 0 Nov 23 16:13:08.645: INFO: coredns-6c7bddbb75-bpdc6 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:08.645: INFO: Container coredns ready: true, restart count 0 Nov 23 16:13:08.645: INFO: inline-volume-tester-mtzn7 started at 2022-11-23 16:12:48 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:08.645: INFO: Container csi-volume-tester ready: true, restart count 0 Nov 23 16:13:08.645: INFO: csi-hostpathplugin-0 started at 2022-11-23 16:12:44 +0000 UTC (0+7 container statuses recorded) Nov 23 16:13:08.645: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:13:08.645: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:13:08.645: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:13:08.645: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 23 16:13:08.645: INFO: Container hostpath ready: true, restart count 0 Nov 23 16:13:08.645: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:13:08.645: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:13:08.645: INFO: netserver-3 started at 2022-11-23 16:12:47 +0000 UTC (0+1 container statuses recorded) Nov 23 16:13:08.645: INFO: Container webserver ready: true, restart count 0 Nov 23 16:13:08.934: INFO: Latency metrics for node i-0fe7a5031fb82cc5b Nov 23 16:13:08.934: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP:�[0m Destroying namespace "volume-expand-5378" for this suite. �[38;5;243m11/23/22 16:13:09.003�[0m
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\(allowExpansion\)\]\svolume\-expand\sshould\sresize\svolume\swhen\sPVC\sis\sedited\swhile\spod\sis\susing\sit$'
test/e2e/storage/testsuites/volume_expand.go:274 k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func5() test/e2e/storage/testsuites/volume_expand.go:274 +0x3b3from junit_01.xml
{"msg":"FAILED External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand should resize volume when PVC is edited while pod is using it","completed":15,"skipped":178,"failed":2,"failures":["External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with mount options","External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand should resize volume when PVC is edited while pod is using it"]} [BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand test/e2e/framework/framework.go:186 �[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m11/23/22 16:19:52.867�[0m Nov 23 16:19:52.867: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP:�[0m Building a namespace api object, basename volume-expand �[38;5;243m11/23/22 16:19:52.868�[0m �[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m11/23/22 16:19:53.068�[0m �[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m11/23/22 16:19:53.2�[0m [It] should resize volume when PVC is edited while pod is using it test/e2e/storage/testsuites/volume_expand.go:252 Nov 23 16:19:53.331: INFO: Creating resource for dynamic PV Nov 23 16:19:53.331: INFO: Using claimSize:1Gi, test suite supported size:{ 1Gi}, driver(ebs.csi.aws.com) supported size:{ 1Gi} �[1mSTEP:�[0m creating a StorageClass volume-expand-1426-e2e-scjhwvr �[38;5;243m11/23/22 16:19:53.331�[0m �[1mSTEP:�[0m creating a claim �[38;5;243m11/23/22 16:19:53.399�[0m Nov 23 16:19:53.399: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP:�[0m Creating a pod with dynamically provisioned volume �[38;5;243m11/23/22 16:19:53.533�[0m Nov 23 16:19:53.601: INFO: Waiting up to 5m0s for pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71" in namespace "volume-expand-1426" to be "running" Nov 23 16:19:53.667: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 66.447072ms Nov 23 16:19:55.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2.133539952s Nov 23 16:19:57.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4.133639283s Nov 23 16:19:59.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 6.132964418s Nov 23 16:20:01.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 8.133730518s Nov 23 16:20:03.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 10.133390507s Nov 23 16:20:05.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 12.133341683s Nov 23 16:20:07.735: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 14.134676613s Nov 23 16:20:09.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 16.133176044s Nov 23 16:20:11.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 18.133610157s Nov 23 16:20:13.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 20.133201718s Nov 23 16:20:15.735: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 22.133855362s Nov 23 16:20:17.735: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 24.134022675s Nov 23 16:20:19.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 26.133691032s Nov 23 16:20:21.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 28.13288507s Nov 23 16:20:23.735: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 30.134266856s Nov 23 16:20:25.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 32.132766018s Nov 23 16:20:27.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 34.133619832s Nov 23 16:20:29.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 36.133112797s Nov 23 16:20:31.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 38.133441617s Nov 23 16:20:33.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 40.133663906s Nov 23 16:20:35.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 42.133612276s Nov 23 16:20:37.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 44.133482065s Nov 23 16:20:39.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 46.133377575s Nov 23 16:20:41.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 48.133549575s Nov 23 16:20:43.737: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 50.136203243s Nov 23 16:20:45.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 52.133169879s Nov 23 16:20:47.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 54.133638467s Nov 23 16:20:49.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 56.13302263s Nov 23 16:20:51.737: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 58.136243601s Nov 23 16:20:53.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.132721243s Nov 23 16:20:55.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.132410528s Nov 23 16:20:57.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.132017667s Nov 23 16:20:59.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.132967126s Nov 23 16:21:01.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.132430148s Nov 23 16:21:03.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.132573382s Nov 23 16:21:05.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.132576692s Nov 23 16:21:07.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.132937537s Nov 23 16:21:09.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.132745459s Nov 23 16:21:11.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.132832669s Nov 23 16:21:13.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.132971167s Nov 23 16:21:15.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.132822745s Nov 23 16:21:17.735: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.134030925s Nov 23 16:21:19.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.132791985s Nov 23 16:21:21.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.132475437s Nov 23 16:21:23.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.132409789s Nov 23 16:21:25.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.132358028s Nov 23 16:21:27.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.131787857s Nov 23 16:21:29.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.132115065s Nov 23 16:21:31.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m38.132704482s Nov 23 16:21:33.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m40.131942639s Nov 23 16:21:35.736: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m42.135090074s Nov 23 16:21:37.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.133751152s Nov 23 16:21:39.736: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.134781481s Nov 23 16:21:41.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.132033596s Nov 23 16:21:43.735: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m50.134396023s Nov 23 16:21:45.732: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m52.131568971s Nov 23 16:21:47.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m54.132395533s Nov 23 16:21:49.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m56.132895145s Nov 23 16:21:51.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 1m58.132944313s Nov 23 16:21:53.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m0.131831186s Nov 23 16:21:55.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m2.132078776s Nov 23 16:21:57.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m4.132331822s Nov 23 16:21:59.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.131932165s Nov 23 16:22:01.732: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.131670054s Nov 23 16:22:03.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.13217312s Nov 23 16:22:05.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.132191322s Nov 23 16:22:07.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.132246541s Nov 23 16:22:09.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.132260614s Nov 23 16:22:11.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.132040535s Nov 23 16:22:13.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.132576012s Nov 23 16:22:15.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.132012435s Nov 23 16:22:17.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.13273238s Nov 23 16:22:19.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.133018834s Nov 23 16:22:21.741: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.140095206s Nov 23 16:22:23.732: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.131659641s Nov 23 16:22:25.736: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.134909868s Nov 23 16:22:27.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.132335299s Nov 23 16:22:29.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.132080463s Nov 23 16:22:31.732: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.131578001s Nov 23 16:22:33.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.132283547s Nov 23 16:22:35.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.132069149s Nov 23 16:22:37.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.133583754s Nov 23 16:22:39.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.131813372s Nov 23 16:22:41.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.132267118s Nov 23 16:22:43.732: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.131498414s Nov 23 16:22:45.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.132065272s Nov 23 16:22:47.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.132396523s Nov 23 16:22:49.735: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.134616537s Nov 23 16:22:51.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 2m58.132363885s Nov 23 16:22:53.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m0.132724806s Nov 23 16:22:55.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m2.13273556s Nov 23 16:22:57.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.132312007s Nov 23 16:22:59.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m6.132224843s Nov 23 16:23:01.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m8.131917615s Nov 23 16:23:03.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.132197023s Nov 23 16:23:05.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.131978677s Nov 23 16:23:07.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m14.132573007s Nov 23 16:23:09.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m16.131887729s Nov 23 16:23:11.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m18.132370591s Nov 23 16:23:13.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m20.132118251s Nov 23 16:23:15.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m22.131928696s Nov 23 16:23:17.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.133762412s Nov 23 16:23:19.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.132372718s Nov 23 16:23:21.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.132326812s Nov 23 16:23:23.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.131926155s Nov 23 16:23:25.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.132706625s Nov 23 16:23:27.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m34.132422761s Nov 23 16:23:29.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m36.132209788s Nov 23 16:23:31.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m38.132343246s Nov 23 16:23:33.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m40.133237233s Nov 23 16:23:35.743: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m42.142468718s Nov 23 16:23:37.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m44.132378896s Nov 23 16:23:39.738: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m46.137351436s Nov 23 16:23:41.732: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m48.131735007s Nov 23 16:23:43.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m50.132898833s Nov 23 16:23:45.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m52.132476075s Nov 23 16:23:47.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m54.132668913s Nov 23 16:23:49.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m56.132125696s Nov 23 16:23:51.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 3m58.132478827s Nov 23 16:23:53.746: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m0.145412893s Nov 23 16:23:55.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m2.132218765s Nov 23 16:23:57.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m4.132339091s Nov 23 16:23:59.735: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.134154564s Nov 23 16:24:01.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.131778239s Nov 23 16:24:03.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.132197248s Nov 23 16:24:05.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.132037427s Nov 23 16:24:07.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.133510226s Nov 23 16:24:09.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m16.132000962s Nov 23 16:24:11.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m18.132224623s Nov 23 16:24:13.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m20.13213044s Nov 23 16:24:15.732: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m22.131748411s Nov 23 16:24:17.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m24.132254827s Nov 23 16:24:19.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.131991963s Nov 23 16:24:21.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.133398791s Nov 23 16:24:23.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.132395196s Nov 23 16:24:25.732: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.131747512s Nov 23 16:24:27.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.132239359s Nov 23 16:24:29.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.132294098s Nov 23 16:24:31.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.132701111s Nov 23 16:24:33.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.13236978s Nov 23 16:24:35.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.132118097s Nov 23 16:24:37.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.132403065s Nov 23 16:24:39.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.132148795s Nov 23 16:24:41.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.132581491s Nov 23 16:24:43.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.133345928s Nov 23 16:24:45.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.132005029s Nov 23 16:24:47.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.132788311s Nov 23 16:24:49.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.132208171s Nov 23 16:24:51.734: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.133046475s Nov 23 16:24:53.733: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.132255712s Nov 23 16:24:53.798: INFO: Pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.197481169s Nov 23 16:24:53.798: INFO: Unexpected error: While creating pods for resizing: <*errors.errorString | 0xc001400940>: { s: "pod \"pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71\" is not Running: timed out while waiting for pod volume-expand-1426/pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71 to be running", } Nov 23 16:24:53.798: FAIL: While creating pods for resizing: pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71" is not Running: timed out while waiting for pod volume-expand-1426/pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71 to be running Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func5() test/e2e/storage/testsuites/volume_expand.go:274 +0x3b3 Nov 23 16:24:53.799: INFO: Deleting pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71" in namespace "volume-expand-1426" Nov 23 16:24:53.869: INFO: Wait up to 5m0s for pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71" to be fully deleted �[1mSTEP:�[0m Deleting pod �[38;5;243m11/23/22 16:24:53.936�[0m Nov 23 16:24:53.936: INFO: Deleting pod "pod-0d2d57df-5c29-4a32-873a-a2cfaf70dc71" in namespace "volume-expand-1426" �[1mSTEP:�[0m Deleting pvc �[38;5;243m11/23/22 16:24:54.001�[0m Nov 23 16:24:54.066: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comhqrbh" �[1mSTEP:�[0m Deleting sc �[38;5;243m11/23/22 16:24:54.132�[0m [AfterEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand test/e2e/framework/framework.go:187 �[1mSTEP:�[0m Collecting events from namespace "volume-expand-1426". �[38;5;243m11/23/22 16:24:54.199�[0m �[1mSTEP:�[0m Found 5 events. �[38;5;243m11/23/22 16:24:54.265�[0m Nov 23 16:24:54.265: INFO: At 2022-11-23 16:19:53 +0000 UTC - event for ebs.csi.aws.comhqrbh: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 16:24:54.265: INFO: At 2022-11-23 16:19:53 +0000 UTC - event for ebs.csi.aws.comhqrbh: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 16:24:54.265: INFO: At 2022-11-23 16:19:53 +0000 UTC - event for ebs.csi.aws.comhqrbh: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-1426/ebs.csi.aws.comhqrbh" Nov 23 16:24:54.265: INFO: At 2022-11-23 16:20:03 +0000 UTC - event for ebs.csi.aws.comhqrbh: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-1426-e2e-scjhwvr": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 16:24:54.265: INFO: At 2022-11-23 16:20:36 +0000 UTC - event for ebs.csi.aws.comhqrbh: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-1426-e2e-scjhwvr": rpc error: code = Internal desc = Could not create volume "pvc-30c38aff-1b7e-43a9-a25f-73265c589f37": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Nov 23 16:24:54.330: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 16:24:54.330: INFO: Nov 23 16:24:54.396: INFO: Logging node info for node i-060d86c6ffc5622b0 Nov 23 16:24:54.461: INFO: Node Info: &Node{ObjectMeta:{i-060d86c6ffc5622b0 b49ff0b3-8686-4c0f-b991-1d427ca43e2a 27298 0 2022-11-23 16:02:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-060d86c6ffc5622b0 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.40.57 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-060d86c6ffc5622b0"} io.cilium.network.ipv4-cilium-host:172.20.57.204 io.cilium.network.ipv4-health-ip:172.20.37.200 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 16:02:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2022-11-23 16:03:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:03:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-060d86c6ffc5622b0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:04:42 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:03:24 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.57,},NodeAddress{Type:ExternalIP,Address:54.200.161.23,},NodeAddress{Type:InternalDNS,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-200-161-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec226e0d77c347467223a292b33eb0c5,SystemUUID:ec226e0d-77c3-4746-7223-a292b33eb0c5,BootID:936b9dd2-17e7-49ac-8c1e-7556bf2c9ee8,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.25.4],SizeBytes:129085904,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.25.4],SizeBytes:118436153,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.25.4],SizeBytes:51921208,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42928334,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41477333,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:85d3f1e9dacc72531445989bb10999e1e70ebc409d11be57e5baa5f031a893b0 registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1],SizeBytes:18257577,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955428,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:24:54.461: INFO: Logging kubelet events for node i-060d86c6ffc5622b0 Nov 23 16:24:54.529: INFO: Logging pods the kubelet thinks is on node i-060d86c6ffc5622b0 Nov 23 16:24:54.601: INFO: cilium-f2gg7 started at 2022-11-23 16:03:06 +0000 UTC (1+1 container statuses recorded) Nov 23 16:24:54.601: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:24:54.601: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 16:24:54.601: INFO: cilium-operator-7f7bcdf787-xnz2q started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:54.601: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 16:24:54.601: INFO: aws-cloud-controller-manager-dzzkp started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:54.601: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Nov 23 16:24:54.601: INFO: etcd-manager-main-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:54.601: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 16:24:54.601: INFO: kube-apiserver-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+2 container statuses recorded) Nov 23 16:24:54.601: INFO: Container healthcheck ready: true, restart count 0 Nov 23 16:24:54.601: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 16:24:54.601: INFO: kube-controller-manager-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:54.601: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 16:24:54.601: INFO: kube-scheduler-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:54.601: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 16:24:54.601: INFO: dns-controller-89c6d5f67-9hm78 started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:54.601: INFO: Container dns-controller ready: true, restart count 0 Nov 23 16:24:54.601: INFO: ebs-csi-node-vpv4t started at 2022-11-23 16:03:06 +0000 UTC (0+3 container statuses recorded) Nov 23 16:24:54.601: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:24:54.601: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:24:54.601: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:24:54.601: INFO: etcd-manager-events-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:54.601: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 16:24:54.601: INFO: kops-controller-7w98n started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:54.601: INFO: Container kops-controller ready: true, restart count 0 Nov 23 16:24:54.855: INFO: Latency metrics for node i-060d86c6ffc5622b0 Nov 23 16:24:54.855: INFO: Logging node info for node i-0765e732a9635923b Nov 23 16:24:54.920: INFO: Node Info: &Node{ObjectMeta:{i-0765e732a9635923b 10981661-cc27-412a-9e66-7a37930b898f 33424 0 2022-11-23 16:04:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0765e732a9635923b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0765e732a9635923b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.58.74 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0765e732a9635923b"} io.cilium.network.ipv4-cilium-host:172.20.54.198 io.cilium.network.ipv4-health-ip:172.20.60.24 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:07 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:24:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0765e732a9635923b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:07 +0000 UTC,LastTransitionTime:2022-11-23 16:05:07 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:24:11 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:24:11 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:24:11 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:24:11 +0000 UTC,LastTransitionTime:2022-11-23 16:05:00 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.58.74,},NodeAddress{Type:ExternalIP,Address:52.25.125.23,},NodeAddress{Type:InternalDNS,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-25-125-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec23d129b4036d8d979053444dfae008,SystemUUID:ec23d129-b403-6d8d-9790-53444dfae008,BootID:fe390540-4601-456e-9686-93a36ea3dc8f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:24:54.920: INFO: Logging kubelet events for node i-0765e732a9635923b Nov 23 16:24:54.988: INFO: Logging pods the kubelet thinks is on node i-0765e732a9635923b Nov 23 16:24:55.058: INFO: test-rolling-update-with-lb-8cc597cbd-n6wxv started at 2022-11-23 16:24:15 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:55.058: INFO: Container agnhost ready: true, restart count 0 Nov 23 16:24:55.058: INFO: hostexec-i-0765e732a9635923b-svg6x started at 2022-11-23 16:24:44 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:55.058: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 16:24:55.058: INFO: cilium-czwdq started at 2022-11-23 16:04:40 +0000 UTC (1+1 container statuses recorded) Nov 23 16:24:55.058: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:24:55.058: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:24:55.058: INFO: local-injector started at 2022-11-23 16:24:53 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:55.058: INFO: Container local-injector ready: false, restart count 0 Nov 23 16:24:55.058: INFO: coredns-6c7bddbb75-2n2dw started at 2022-11-23 16:05:19 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:55.058: INFO: Container coredns ready: true, restart count 0 Nov 23 16:24:55.058: INFO: run-test-3 started at 2022-11-23 16:24:21 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:55.058: INFO: Container run-test-3 ready: true, restart count 0 Nov 23 16:24:55.058: INFO: exec-volume-test-preprovisionedpv-pp8m started at 2022-11-23 16:24:52 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:55.058: INFO: Container exec-container-preprovisionedpv-pp8m ready: false, restart count 0 Nov 23 16:24:55.058: INFO: ebs-csi-node-5khlc started at 2022-11-23 16:04:40 +0000 UTC (0+3 container statuses recorded) Nov 23 16:24:55.058: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:24:55.058: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:24:55.058: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:24:55.058: INFO: hostexec-i-0765e732a9635923b-5t57w started at 2022-11-23 16:24:35 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:55.058: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 16:24:55.058: INFO: pod-hostip-9149145d-9265-4f59-9a7c-d3536061c10b started at 2022-11-23 16:24:50 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:55.058: INFO: Container test ready: true, restart count 0 Nov 23 16:24:55.332: INFO: Latency metrics for node i-0765e732a9635923b Nov 23 16:24:55.332: INFO: Logging node info for node i-0ade3c3082581cb52 Nov 23 16:24:55.397: INFO: Node Info: &Node{ObjectMeta:{i-0ade3c3082581cb52 2305662a-7d05-4979-bcec-220b8b4672db 33962 0 2022-11-23 16:04:42 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0ade3c3082581cb52 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0ade3c3082581cb52 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.56.12 csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-753":"i-0ade3c3082581cb52","ebs.csi.aws.com":"i-0ade3c3082581cb52"} io.cilium.network.ipv4-cilium-host:172.20.45.237 io.cilium.network.ipv4-health-ip:172.20.55.90 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:24:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {kube-controller-manager Update v1 2022-11-23 16:24:49 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0ade3c3082581cb52,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:12 +0000 UTC,LastTransitionTime:2022-11-23 16:05:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:24:05 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:24:05 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:24:05 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:24:05 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.56.12,},NodeAddress{Type:ExternalIP,Address:35.92.243.169,},NodeAddress{Type:InternalDNS,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-92-243-169.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bd980dfb52643cb4857d57b45824b,SystemUUID:ec2bd980-dfb5-2643-cb48-57d57b45824b,BootID:55912341-dce8-4199-a57c-c8703e24af46,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 registry.k8s.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-753^59016781-6b4b-11ed-9580-5e56fabb9126,DevicePath:,},},Config:nil,},} Nov 23 16:24:55.398: INFO: Logging kubelet events for node i-0ade3c3082581cb52 Nov 23 16:24:55.465: INFO: Logging pods the kubelet thinks is on node i-0ade3c3082581cb52 Nov 23 16:24:55.535: INFO: pvc-volume-tester-7vpz4 started at 2022-11-23 16:24:48 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:55.535: INFO: Container volume-tester ready: false, restart count 0 Nov 23 16:24:55.535: INFO: csi-mockplugin-0 started at 2022-11-23 16:24:37 +0000 UTC (0+3 container statuses recorded) Nov 23 16:24:55.535: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:24:55.535: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 16:24:55.535: INFO: Container mock ready: true, restart count 0 Nov 23 16:24:55.535: INFO: test-rolling-update-with-lb-8cc597cbd-zktkh started at 2022-11-23 16:24:11 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:55.535: INFO: Container agnhost ready: true, restart count 0 Nov 23 16:24:55.535: INFO: cilium-q5tsk started at 2022-11-23 16:04:43 +0000 UTC (1+1 container statuses recorded) Nov 23 16:24:55.535: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:24:55.535: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:24:55.535: INFO: ebs-csi-node-t56sd started at 2022-11-23 16:04:43 +0000 UTC (0+3 container statuses recorded) Nov 23 16:24:55.535: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:24:55.535: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:24:55.535: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:24:55.535: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 16:24:37 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:55.535: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:24:55.845: INFO: Latency metrics for node i-0ade3c3082581cb52 Nov 23 16:24:55.845: INFO: Logging node info for node i-0bf2756d84bae9168 Nov 23 16:24:55.911: INFO: Node Info: &Node{ObjectMeta:{i-0bf2756d84bae9168 2f26f127-5eb0-44f3-97f0-990045e7b900 27933 0 2022-11-23 16:04:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0bf2756d84bae9168 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0bf2756d84bae9168 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.218 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0bf2756d84bae9168"} io.cilium.network.ipv4-cilium-host:172.20.35.201 io.cilium.network.ipv4-health-ip:172.20.51.170 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:02 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:20:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0bf2756d84bae9168,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:02 +0000 UTC,LastTransitionTime:2022-11-23 16:05:02 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.218,},NodeAddress{Type:ExternalIP,Address:34.221.37.130,},NodeAddress{Type:InternalDNS,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-37-130.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d7db7f0ff759b7a6209e832f1dcea,SystemUUID:ec2d7db7-f0ff-759b-7a62-09e832f1dcea,BootID:8f294a85-a1ed-4dfb-bdb9-f8818e1f067f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f registry.k8s.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:24:55.911: INFO: Logging kubelet events for node i-0bf2756d84bae9168 Nov 23 16:24:55.979: INFO: Logging pods the kubelet thinks is on node i-0bf2756d84bae9168 Nov 23 16:24:56.049: INFO: startup-19a184f5-ddfb-4fc3-855b-ca2d08482379 started at 2022-11-23 16:23:57 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:56.049: INFO: Container busybox ready: false, restart count 0 Nov 23 16:24:56.049: INFO: dns-test-394a8bc1-4d78-428a-a69e-beb002351789 started at 2022-11-23 16:24:50 +0000 UTC (0+3 container statuses recorded) Nov 23 16:24:56.049: INFO: Container jessie-querier ready: false, restart count 0 Nov 23 16:24:56.049: INFO: Container querier ready: false, restart count 0 Nov 23 16:24:56.049: INFO: Container webserver ready: false, restart count 0 Nov 23 16:24:56.049: INFO: ebs-csi-controller-847c4b8bc4-kwpsb started at 2022-11-23 16:04:56 +0000 UTC (0+5 container statuses recorded) Nov 23 16:24:56.049: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:24:56.049: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:24:56.049: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:24:56.049: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:24:56.049: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:24:56.049: INFO: test-rolling-update-with-lb-8cc597cbd-lsvgw started at 2022-11-23 16:24:13 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:56.049: INFO: Container agnhost ready: true, restart count 0 Nov 23 16:24:56.049: INFO: cilium-scm24 started at 2022-11-23 16:04:35 +0000 UTC (1+1 container statuses recorded) Nov 23 16:24:56.049: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:24:56.049: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:24:56.049: INFO: ebs-csi-node-crshz started at 2022-11-23 16:04:35 +0000 UTC (0+3 container statuses recorded) Nov 23 16:24:56.049: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:24:56.049: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:24:56.049: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:24:56.049: INFO: logs-generator started at 2022-11-23 16:24:53 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:56.049: INFO: Container logs-generator ready: true, restart count 0 Nov 23 16:24:56.351: INFO: Latency metrics for node i-0bf2756d84bae9168 Nov 23 16:24:56.351: INFO: Logging node info for node i-0fe7a5031fb82cc5b Nov 23 16:24:56.416: INFO: Node Info: &Node{ObjectMeta:{i-0fe7a5031fb82cc5b 0297efe3-5070-45f1-8aa2-9a1056b17b59 33516 0 2022-11-23 16:04:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0fe7a5031fb82cc5b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0fe7a5031fb82cc5b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.89 csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-645":"i-0fe7a5031fb82cc5b","ebs.csi.aws.com":"i-0fe7a5031fb82cc5b"} io.cilium.network.ipv4-cilium-host:172.20.57.57 io.cilium.network.ipv4-health-ip:172.20.59.99 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-11-23 16:04:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:03 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:24:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0fe7a5031fb82cc5b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:03 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:23:58 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:23:58 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:23:58 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:23:58 +0000 UTC,LastTransitionTime:2022-11-23 16:04:54 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.89,},NodeAddress{Type:ExternalIP,Address:34.221.99.167,},NodeAddress{Type:InternalDNS,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-99-167.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29292cb2c053c780af3b39ea2fa8c6,SystemUUID:ec29292c-b2c0-53c7-80af-3b39ea2fa8c6,BootID:ff4df73e-ccb2-4109-99ca-04a1b7ab5611,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 16:24:56.417: INFO: Logging kubelet events for node i-0fe7a5031fb82cc5b Nov 23 16:24:56.486: INFO: Logging pods the kubelet thinks is on node i-0fe7a5031fb82cc5b Nov 23 16:24:56.556: INFO: pvc-volume-tester-h2rrj started at 2022-11-23 16:24:29 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:56.556: INFO: Container volume-tester ready: true, restart count 0 Nov 23 16:24:56.556: INFO: ebs-csi-node-8ffwl started at 2022-11-23 16:04:34 +0000 UTC (0+3 container statuses recorded) Nov 23 16:24:56.556: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:24:56.556: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:24:56.556: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 16:24:56.556: INFO: cilium-727pv started at 2022-11-23 16:04:34 +0000 UTC (1+1 container statuses recorded) Nov 23 16:24:56.556: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 16:24:56.556: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 16:24:56.556: INFO: coredns-6c7bddbb75-bpdc6 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:56.556: INFO: Container coredns ready: true, restart count 0 Nov 23 16:24:56.557: INFO: csi-mockplugin-0 started at 2022-11-23 16:24:18 +0000 UTC (0+3 container statuses recorded) Nov 23 16:24:56.557: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:24:56.557: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 16:24:56.557: INFO: Container mock ready: true, restart count 0 Nov 23 16:24:56.557: INFO: coredns-autoscaler-db7b97744-r4jm7 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded) Nov 23 16:24:56.557: INFO: Container autoscaler ready: true, restart count 0 Nov 23 16:24:56.557: INFO: ebs-csi-controller-847c4b8bc4-lxdkr started at 2022-11-23 16:04:54 +0000 UTC (0+5 container statuses recorded) Nov 23 16:24:56.557: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 16:24:56.557: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 16:24:56.557: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 16:24:56.557: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 16:24:56.557: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 16:24:56.861: INFO: Latency metrics for node i-0fe7a5031fb82cc5b Nov 23 16:24:56.861: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP:�[0m Destroying namespace "volume-expand-1426" for this suite. �[38;5;243m11/23/22 16:24:56.927�[0m
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(Always\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\snew\spod\sfsgroup\sapplied\sto\svolume\scontents$'
test/e2e/storage/testsuites/fsgroupchangepolicy.go:264
k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc0013f3b80, 0xc003439e30, 0x1, {0x0, 0x0}, {0x0, 0x0})
test/e2e/storage/testsuites/fsgroupchangepolicy.go:264 +0x151
k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3()
test/e2e/storage/testsuites/fsgroupchangepolicy.go:235 +0x298
{"msg":"FAILED External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","completed":5,"skipped":49,"failed":2,"failures":["External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents"]}
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
test/e2e/framework/framework.go:186
�[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m11/23/22 16:16:58.92�[0m
Nov 23 16:16:58.920: INFO: >>> kubeConfig: /root/.kube/config
�[1mSTEP:�[0m Building a namespace api object, basename fsgroupchangepolicy �[38;5;243m11/23/22 16:16:58.921�[0m
�[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m11/23/22 16:16:59.111�[0m
�[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m11/23/22 16:16:59.236�[0m
[It] (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
Nov 23 16:16:59.361: INFO: Creating resource for dynamic PV
Nov 23 16:16:59.361: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi}
�[1mSTEP:�[0m creating a StorageClass fsgroupchangepolicy-801-e2e-scm9zfr �[38;5;243m11/23/22 16:16:59.361�[0m
�[1mSTEP:�[0m creating a claim �[38;5;243m11/23/22 16:16:59.425�[0m
Nov 23 16:16:59.425: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
�[1mSTEP:�[0m Creating Pod in namespace fsgroupchangepolicy-801 with fsgroup 1000 �[38;5;243m11/23/22 16:16:59.553�[0m
Nov 23 16:16:59.619: INFO: Waiting up to 5m0s for pod "pod-9bd31862-4856-412b-9df4-40099a9573c7" in namespace "fsgroupchangepolicy-801" to be "running"
Nov 23 16:16:59.682: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 63.011354ms
Nov 23 16:17:01.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2.127127787s
Nov 23 16:17:03.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4.126890568s
Nov 23 16:17:05.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 6.126649879s
Nov 23 16:17:07.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 8.126738543s
Nov 23 16:17:09.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 10.126950131s
Nov 23 16:17:11.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 12.126221449s
Nov 23 16:17:13.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 14.126418503s
Nov 23 16:17:15.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 16.126128341s
Nov 23 16:17:17.749: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 18.1299147s
Nov 23 16:17:19.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 20.127301397s
Nov 23 16:17:21.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 22.127144648s
Nov 23 16:17:23.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 24.126488403s
Nov 23 16:17:25.748: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 26.128631316s
Nov 23 16:17:27.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 28.126696565s
Nov 23 16:17:29.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 30.127221713s
Nov 23 16:17:31.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 32.126756548s
Nov 23 16:17:33.754: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 34.135215606s
Nov 23 16:17:35.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 36.127061409s
Nov 23 16:17:37.747: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 38.127648202s
Nov 23 16:17:39.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 40.126909724s
Nov 23 16:17:41.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 42.12653469s
Nov 23 16:17:43.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 44.126133452s
Nov 23 16:17:45.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 46.126514804s
Nov 23 16:17:47.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 48.126324872s
Nov 23 16:17:49.749: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 50.129786395s
Nov 23 16:17:51.748: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 52.128986491s
Nov 23 16:17:53.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 54.126889411s
Nov 23 16:17:55.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 56.126779581s
Nov 23 16:17:57.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 58.126344652s
Nov 23 16:17:59.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.126632863s
Nov 23 16:18:01.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.127135987s
Nov 23 16:18:03.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.126769412s
Nov 23 16:18:05.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.126252885s
Nov 23 16:18:07.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.1264049s
Nov 23 16:18:09.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.126405988s
Nov 23 16:18:11.747: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.127741462s
Nov 23 16:18:13.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.126745646s
Nov 23 16:18:15.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.127103552s
Nov 23 16:18:17.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.126815125s
Nov 23 16:18:19.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.12634016s
Nov 23 16:18:21.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.126764817s
Nov 23 16:18:23.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.127220782s
Nov 23 16:18:25.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.126359451s
Nov 23 16:18:27.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.127133157s
Nov 23 16:18:29.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.126114959s
Nov 23 16:18:31.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.127132305s
Nov 23 16:18:33.747: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.12773083s
Nov 23 16:18:35.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.126757524s
Nov 23 16:18:37.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m38.126682822s
Nov 23 16:18:39.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m40.126895743s
Nov 23 16:18:41.747: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m42.127741066s
Nov 23 16:18:43.747: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.127709829s
Nov 23 16:18:45.747: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.128274826s
Nov 23 16:18:47.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.126739364s
Nov 23 16:18:49.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m50.126568792s
Nov 23 16:18:51.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m52.127384019s
Nov 23 16:18:53.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m54.126555856s
Nov 23 16:18:55.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m56.127390802s
Nov 23 16:18:57.747: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m58.128318788s
Nov 23 16:18:59.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m0.126639771s
Nov 23 16:19:01.753: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m2.13423913s
Nov 23 16:19:03.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m4.126160936s
Nov 23 16:19:05.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.126783756s
Nov 23 16:19:07.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.126741415s
Nov 23 16:19:09.758: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.139218688s
Nov 23 16:19:11.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.126606358s
Nov 23 16:19:13.747: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.128345089s
Nov 23 16:19:15.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.126476023s
Nov 23 16:19:17.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.1269338s
Nov 23 16:19:19.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.127051467s
Nov 23 16:19:21.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.127082968s
Nov 23 16:19:23.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.127151162s
Nov 23 16:19:25.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.127449792s
Nov 23 16:19:27.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.127348699s
Nov 23 16:19:29.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.127353911s
Nov 23 16:19:31.747: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.127742057s
Nov 23 16:19:33.747: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.128450555s
Nov 23 16:19:35.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.126667377s
Nov 23 16:19:37.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.126510848s
Nov 23 16:19:39.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.127158657s
Nov 23 16:19:41.747: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.127703669s
Nov 23 16:19:43.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.126811518s
Nov 23 16:19:45.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.126583606s
Nov 23 16:19:47.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.126505081s
Nov 23 16:19:49.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.127070089s
Nov 23 16:19:51.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.127289314s
Nov 23 16:19:53.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.126969772s
Nov 23 16:19:55.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.126685045s
Nov 23 16:19:57.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 2m58.126879047s
Nov 23 16:19:59.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m0.126898016s
Nov 23 16:20:01.747: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m2.128240891s
Nov 23 16:20:03.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.126931446s
Nov 23 16:20:05.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m6.12642453s
Nov 23 16:20:07.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m8.127309405s
Nov 23 16:20:09.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.126247395s
Nov 23 16:20:11.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.126535657s
Nov 23 16:20:13.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m14.12699981s
Nov 23 16:20:15.747: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m16.128201144s
Nov 23 16:20:17.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m18.127117496s
Nov 23 16:20:19.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m20.126975617s
Nov 23 16:20:21.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m22.126716956s
Nov 23 16:20:23.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.127126287s
Nov 23 16:20:25.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.126284385s
Nov 23 16:20:27.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.125939508s
Nov 23 16:20:29.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.127095213s
Nov 23 16:20:31.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.126379337s
Nov 23 16:20:33.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m34.126779741s
Nov 23 16:20:35.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m36.127174058s
Nov 23 16:20:37.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m38.126029653s
Nov 23 16:20:39.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m40.126763616s
Nov 23 16:20:41.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m42.126513544s
Nov 23 16:20:43.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m44.126481103s
Nov 23 16:20:45.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m46.12625755s
Nov 23 16:20:47.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m48.126304396s
Nov 23 16:20:49.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m50.127232195s
Nov 23 16:20:51.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m52.12685162s
Nov 23 16:20:53.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m54.126835967s
Nov 23 16:20:55.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m56.127019893s
Nov 23 16:20:57.747: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 3m58.128358592s
Nov 23 16:20:59.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m0.126829169s
Nov 23 16:21:01.749: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m2.129685995s
Nov 23 16:21:03.750: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m4.131186534s
Nov 23 16:21:05.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.126879065s
Nov 23 16:21:07.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.126858182s
Nov 23 16:21:09.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.127249671s
Nov 23 16:21:11.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.127054198s
Nov 23 16:21:13.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.127169694s
Nov 23 16:21:15.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m16.127311181s
Nov 23 16:21:17.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m18.126266886s
Nov 23 16:21:19.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m20.127006429s
Nov 23 16:21:21.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m22.12676325s
Nov 23 16:21:23.747: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m24.128351352s
Nov 23 16:21:25.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.126646139s
Nov 23 16:21:27.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.126378737s
Nov 23 16:21:29.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.126816955s
Nov 23 16:21:31.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.12719699s
Nov 23 16:21:33.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.126830331s
Nov 23 16:21:35.748: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.129168347s
Nov 23 16:21:37.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.126739433s
Nov 23 16:21:39.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.126997011s
Nov 23 16:21:41.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.127113889s
Nov 23 16:21:43.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.126871512s
Nov 23 16:21:45.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.126428756s
Nov 23 16:21:47.745: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.12651584s
Nov 23 16:21:49.749: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.130062856s
Nov 23 16:21:51.748: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.128834991s
Nov 23 16:21:53.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.126847839s
Nov 23 16:21:55.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.127065406s
Nov 23 16:21:57.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.127092221s
Nov 23 16:21:59.746: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.126818845s
Nov 23 16:21:59.809: INFO: Pod "pod-9bd31862-4856-412b-9df4-40099a9573c7": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.190169051s
Nov 23 16:21:59.809: INFO: Unexpected error:
<*errors.errorString | 0xc0010c54c0>: {
s: "pod \"pod-9bd31862-4856-412b-9df4-40099a9573c7\" is not Running: timed out while waiting for pod fsgroupchangepolicy-801/pod-9bd31862-4856-412b-9df4-40099a9573c7 to be running",
}
Nov 23 16:21:59.809: FAIL: pod "pod-9bd31862-4856-412b-9df4-40099a9573c7" is not Running: timed out while waiting for pod fsgroupchangepolicy-801/pod-9bd31862-4856-412b-9df4-40099a9573c7 to be running
Full Stack Trace
k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc0013f3b80, 0xc003439e30, 0x1, {0x0, 0x0}, {0x0, 0x0})
test/e2e/storage/testsuites/fsgroupchangepolicy.go:264 +0x151
k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3()
test/e2e/storage/testsuites/fsgroupchangepolicy.go:235 +0x298
�[1mSTEP:�[0m Deleting pvc �[38;5;243m11/23/22 16:21:59.809�[0m
Nov 23 16:21:59.873: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.com7mg27"
�[1mSTEP:�[0m Deleting sc �[38;5;243m11/23/22 16:21:59.937�[0m
[AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
test/e2e/framework/framework.go:187
�[1mSTEP:�[0m Collecting events from namespace "fsgroupchangepolicy-801". �[38;5;243m11/23/22 16:22:00.001�[0m
�[1mSTEP:�[0m Found 5 events. �[38;5;243m11/23/22 16:22:00.064�[0m
Nov 23 16:22:00.065: INFO: At 2022-11-23 16:16:59 +0000 UTC - event for ebs.csi.aws.com7mg27: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
Nov 23 16:22:00.065: INFO: At 2022-11-23 16:16:59 +0000 UTC - event for ebs.csi.aws.com7mg27: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
Nov 23 16:22:00.065: INFO: At 2022-11-23 16:16:59 +0000 UTC - event for ebs.csi.aws.com7mg27: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-801/ebs.csi.aws.com7mg27"
Nov 23 16:22:00.065: INFO: At 2022-11-23 16:17:09 +0000 UTC - event for ebs.csi.aws.com7mg27: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-801-e2e-scm9zfr": rpc error: code = Internal desc = Could not create volume "pvc-23b237b0-5744-47f2-99de-1b099a918825": could not create volume in EC2: RequestCanceled: request context canceled
caused by: context deadline exceeded
Nov 23 16:22:00.065: INFO: At 2022-11-23 16:17:32 +0000 UTC - event for ebs.csi.aws.com7mg27: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-801-e2e-scm9zfr": rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 23 16:22:00.128: INFO: POD NODE PHASE GRACE CONDITIONS
Nov 23 16:22:00.128: INFO: pod-9bd31862-4856-412b-9df4-40099a9573c7 Pending []
Nov 23 16:22:00.128: INFO:
Nov 23 16:22:00.256: INFO:
Logging node info for node i-060d86c6ffc5622b0
Nov 23 16:22:00.320: INFO: Node Info: &Node{ObjectMeta:{i-060d86c6ffc5622b0 b49ff0b3-8686-4c0f-b991-1d427ca43e2a 27298 0 2022-11-23 16:02:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-060d86c6ffc5622b0 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.40.57 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-060d86c6ffc5622b0"} io.cilium.network.ipv4-cilium-host:172.20.57.204 io.cilium.network.ipv4-health-ip:172.20.37.200 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 16:02:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2022-11-23 16:03:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:03:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:20:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-060d86c6ffc5622b0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:04:42 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:20:25 +0000 UTC,LastTransitionTime:2022-11-23 16:03:24 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.57,},NodeAddress{Type:ExternalIP,Address:54.200.161.23,},NodeAddress{Type:InternalDNS,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-200-161-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec226e0d77c347467223a292b33eb0c5,SystemUUID:ec226e0d-77c3-4746-7223-a292b33eb0c5,BootID:936b9dd2-17e7-49ac-8c1e-7556bf2c9ee8,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.25.4],SizeBytes:129085904,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.25.4],SizeBytes:118436153,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.25.4],SizeBytes:51921208,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42928334,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41477333,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:85d3f1e9dacc72531445989bb10999e1e70ebc409d11be57e5baa5f031a893b0 registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1],SizeBytes:18257577,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955428,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov 23 16:22:00.320: INFO:
Logging kubelet events for node i-060d86c6ffc5622b0
Nov 23 16:22:00.386: INFO:
Logging pods the kubelet thinks is on node i-060d86c6ffc5622b0
Nov 23 16:22:00.454: INFO: kube-apiserver-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+2 container statuses recorded)
Nov 23 16:22:00.454: INFO: Container healthcheck ready: true, restart count 0
Nov 23 16:22:00.454: INFO: Container kube-apiserver ready: true, restart count 1
Nov 23 16:22:00.454: INFO: kube-controller-manager-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:00.454: INFO: Container kube-controller-manager ready: true, restart count 2
Nov 23 16:22:00.454: INFO: kube-scheduler-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:00.454: INFO: Container kube-scheduler ready: true, restart count 0
Nov 23 16:22:00.454: INFO: dns-controller-89c6d5f67-9hm78 started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:00.454: INFO: Container dns-controller ready: true, restart count 0
Nov 23 16:22:00.454: INFO: cilium-f2gg7 started at 2022-11-23 16:03:06 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:22:00.454: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:22:00.454: INFO: Container cilium-agent ready: true, restart count 1
Nov 23 16:22:00.454: INFO: cilium-operator-7f7bcdf787-xnz2q started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:00.454: INFO: Container cilium-operator ready: true, restart count 1
Nov 23 16:22:00.454: INFO: aws-cloud-controller-manager-dzzkp started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:00.454: INFO: Container aws-cloud-controller-manager ready: true, restart count 0
Nov 23 16:22:00.454: INFO: etcd-manager-main-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:00.454: INFO: Container etcd-manager ready: true, restart count 0
Nov 23 16:22:00.454: INFO: ebs-csi-node-vpv4t started at 2022-11-23 16:03:06 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:22:00.454: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:22:00.454: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:22:00.454: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:22:00.454: INFO: kops-controller-7w98n started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:00.454: INFO: Container kops-controller ready: true, restart count 0
Nov 23 16:22:00.454: INFO: etcd-manager-events-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:00.454: INFO: Container etcd-manager ready: true, restart count 0
Nov 23 16:22:00.697: INFO:
Latency metrics for node i-060d86c6ffc5622b0
Nov 23 16:22:00.697: INFO:
Logging node info for node i-0765e732a9635923b
Nov 23 16:22:00.761: INFO: Node Info: &Node{ObjectMeta:{i-0765e732a9635923b 10981661-cc27-412a-9e66-7a37930b898f 28980 0 2022-11-23 16:04:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0765e732a9635923b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0765e732a9635923b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.58.74 csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9722":"i-0765e732a9635923b","ebs.csi.aws.com":"i-0765e732a9635923b"} io.cilium.network.ipv4-cilium-host:172.20.54.198 io.cilium.network.ipv4-health-ip:172.20.60.24 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:07 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:21:44 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 16:21:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0765e732a9635923b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:07 +0000 UTC,LastTransitionTime:2022-11-23 16:05:07 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:48 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:48 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:48 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:21:48 +0000 UTC,LastTransitionTime:2022-11-23 16:05:00 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.58.74,},NodeAddress{Type:ExternalIP,Address:52.25.125.23,},NodeAddress{Type:InternalDNS,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-25-125-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec23d129b4036d8d979053444dfae008,SystemUUID:ec23d129-b403-6d8d-9790-53444dfae008,BootID:fe390540-4601-456e-9686-93a36ea3dc8f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-9722^eb42a40e-6b4a-11ed-8178-a6469949b10b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-9722^eb42a40e-6b4a-11ed-8178-a6469949b10b,DevicePath:,},},Config:nil,},}
Nov 23 16:22:00.761: INFO:
Logging kubelet events for node i-0765e732a9635923b
Nov 23 16:22:00.831: INFO:
Logging pods the kubelet thinks is on node i-0765e732a9635923b
Nov 23 16:22:00.908: INFO: ebs-csi-node-5khlc started at 2022-11-23 16:04:40 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:22:00.908: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:22:00.908: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:22:00.908: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:22:00.908: INFO: netserver-0 started at 2022-11-23 16:21:29 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:00.908: INFO: Container webserver ready: true, restart count 0
Nov 23 16:22:00.908: INFO: csi-mockplugin-0 started at 2022-11-23 16:21:38 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:22:00.908: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:22:00.908: INFO: Container driver-registrar ready: true, restart count 0
Nov 23 16:22:00.908: INFO: Container mock ready: true, restart count 0
Nov 23 16:22:00.908: INFO: cilium-czwdq started at 2022-11-23 16:04:40 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:22:00.908: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:22:00.908: INFO: Container cilium-agent ready: true, restart count 0
Nov 23 16:22:00.908: INFO: probe-test-d560ffab-afe1-45a8-940d-fdc4718a3166 started at 2022-11-23 16:21:24 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:00.908: INFO: Container probe-test-d560ffab-afe1-45a8-940d-fdc4718a3166 ready: false, restart count 0
Nov 23 16:22:00.908: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 16:21:38 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:00.908: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:22:00.908: INFO: coredns-6c7bddbb75-2n2dw started at 2022-11-23 16:05:19 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:00.908: INFO: Container coredns ready: true, restart count 0
Nov 23 16:22:00.908: INFO: test-rolling-update-with-lb-dbbf5f9b8-hp5hf started at 2022-11-23 16:21:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:00.908: INFO: Container agnhost ready: true, restart count 0
Nov 23 16:22:00.908: INFO: deployment-97c6466db-trmfq started at 2022-11-23 16:20:52 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:00.908: INFO: Container nginx ready: false, restart count 0
Nov 23 16:22:00.908: INFO: pvc-volume-tester-fb7cs started at 2022-11-23 16:21:43 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:00.908: INFO: Container volume-tester ready: true, restart count 0
Nov 23 16:22:01.196: INFO:
Latency metrics for node i-0765e732a9635923b
Nov 23 16:22:01.196: INFO:
Logging node info for node i-0ade3c3082581cb52
Nov 23 16:22:01.259: INFO: Node Info: &Node{ObjectMeta:{i-0ade3c3082581cb52 2305662a-7d05-4979-bcec-220b8b4672db 29101 0 2022-11-23 16:04:42 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0ade3c3082581cb52 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0ade3c3082581cb52 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.56.12 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-4466":"i-0ade3c3082581cb52","ebs.csi.aws.com":"i-0ade3c3082581cb52"} io.cilium.network.ipv4-cilium-host:172.20.45.237 io.cilium.network.ipv4-health-ip:172.20.55.90 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:21:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0ade3c3082581cb52,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:12 +0000 UTC,LastTransitionTime:2022-11-23 16:05:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:21 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:21 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:21 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:21:21 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.56.12,},NodeAddress{Type:ExternalIP,Address:35.92.243.169,},NodeAddress{Type:InternalDNS,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-92-243-169.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bd980dfb52643cb4857d57b45824b,SystemUUID:ec2bd980-dfb5-2643-cb48-57d57b45824b,BootID:55912341-dce8-4199-a57c-c8703e24af46,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 registry.k8s.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov 23 16:22:01.259: INFO:
Logging kubelet events for node i-0ade3c3082581cb52
Nov 23 16:22:01.331: INFO:
Logging pods the kubelet thinks is on node i-0ade3c3082581cb52
Nov 23 16:22:01.398: INFO: cilium-q5tsk started at 2022-11-23 16:04:43 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:22:01.398: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:22:01.398: INFO: Container cilium-agent ready: true, restart count 0
Nov 23 16:22:01.398: INFO: ebs-csi-node-t56sd started at 2022-11-23 16:04:43 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:22:01.398: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:22:01.398: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:22:01.398: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:22:01.398: INFO: csi-hostpathplugin-0 started at 2022-11-23 16:21:50 +0000 UTC (0+7 container statuses recorded)
Nov 23 16:22:01.398: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:22:01.398: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:22:01.398: INFO: Container csi-resizer ready: true, restart count 0
Nov 23 16:22:01.398: INFO: Container csi-snapshotter ready: true, restart count 0
Nov 23 16:22:01.398: INFO: Container hostpath ready: true, restart count 0
Nov 23 16:22:01.398: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:22:01.398: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:22:01.398: INFO: netserver-1 started at 2022-11-23 16:21:30 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:01.398: INFO: Container webserver ready: true, restart count 0
Nov 23 16:22:01.398: INFO: inline-volume-tester-dc5fg started at 2022-11-23 16:21:50 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:01.398: INFO: Container csi-volume-tester ready: true, restart count 0
Nov 23 16:22:01.398: INFO: test-rolling-update-with-lb-dbbf5f9b8-bhfzj started at 2022-11-23 16:21:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:01.398: INFO: Container agnhost ready: true, restart count 0
Nov 23 16:22:01.679: INFO:
Latency metrics for node i-0ade3c3082581cb52
Nov 23 16:22:01.679: INFO:
Logging node info for node i-0bf2756d84bae9168
Nov 23 16:22:01.743: INFO: Node Info: &Node{ObjectMeta:{i-0bf2756d84bae9168 2f26f127-5eb0-44f3-97f0-990045e7b900 27933 0 2022-11-23 16:04:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0bf2756d84bae9168 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0bf2756d84bae9168 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.218 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0bf2756d84bae9168"} io.cilium.network.ipv4-cilium-host:172.20.35.201 io.cilium.network.ipv4-health-ip:172.20.51.170 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:02 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:20:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0bf2756d84bae9168,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:02 +0000 UTC,LastTransitionTime:2022-11-23 16:05:02 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:20:52 +0000 UTC,LastTransitionTime:2022-11-23 16:04:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.218,},NodeAddress{Type:ExternalIP,Address:34.221.37.130,},NodeAddress{Type:InternalDNS,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-37-130.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d7db7f0ff759b7a6209e832f1dcea,SystemUUID:ec2d7db7-f0ff-759b-7a62-09e832f1dcea,BootID:8f294a85-a1ed-4dfb-bdb9-f8818e1f067f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f registry.k8s.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov 23 16:22:01.743: INFO:
Logging kubelet events for node i-0bf2756d84bae9168
Nov 23 16:22:01.809: INFO:
Logging pods the kubelet thinks is on node i-0bf2756d84bae9168
Nov 23 16:22:01.879: INFO: inline-volume-nvmnf started at 2022-11-23 16:21:52 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:01.879: INFO: Container volume-tester ready: false, restart count 0
Nov 23 16:22:01.879: INFO: test-runtimeclass-runtimeclass-4773-preconfigured-handler-4pfhk started at 2022-11-23 16:21:57 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:01.879: INFO: Container test ready: false, restart count 0
Nov 23 16:22:01.879: INFO: ebs-csi-controller-847c4b8bc4-kwpsb started at 2022-11-23 16:04:56 +0000 UTC (0+5 container statuses recorded)
Nov 23 16:22:01.879: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:22:01.879: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:22:01.879: INFO: Container csi-resizer ready: true, restart count 0
Nov 23 16:22:01.879: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:22:01.879: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:22:01.879: INFO: pod-subpath-test-inlinevolume-wr6x started at 2022-11-23 16:21:58 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:22:01.879: INFO: Init container init-volume-inlinevolume-wr6x ready: true, restart count 0
Nov 23 16:22:01.879: INFO: Container test-container-subpath-inlinevolume-wr6x ready: false, restart count 0
Nov 23 16:22:01.879: INFO: test-container-pod started at 2022-11-23 16:21:52 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:01.879: INFO: Container webserver ready: true, restart count 0
Nov 23 16:22:01.879: INFO: ss-0 started at 2022-11-23 16:22:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:01.879: INFO: Container webserver ready: false, restart count 0
Nov 23 16:22:01.879: INFO: hostexec-i-0bf2756d84bae9168-vtbq2 started at 2022-11-23 16:21:58 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:01.879: INFO: Container agnhost-container ready: true, restart count 0
Nov 23 16:22:01.879: INFO: netserver-2 started at 2022-11-23 16:21:30 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:01.879: INFO: Container webserver ready: true, restart count 0
Nov 23 16:22:01.879: INFO: cilium-scm24 started at 2022-11-23 16:04:35 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:22:01.879: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:22:01.879: INFO: Container cilium-agent ready: true, restart count 0
Nov 23 16:22:01.879: INFO: test-rolling-update-with-lb-dbbf5f9b8-7fpt7 started at 2022-11-23 16:21:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:01.879: INFO: Container agnhost ready: true, restart count 0
Nov 23 16:22:01.879: INFO: ebs-csi-node-crshz started at 2022-11-23 16:04:35 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:22:01.879: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:22:01.879: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:22:01.879: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:22:02.269: INFO:
Latency metrics for node i-0bf2756d84bae9168
Nov 23 16:22:02.269: INFO:
Logging node info for node i-0fe7a5031fb82cc5b
Nov 23 16:22:02.332: INFO: Node Info: &Node{ObjectMeta:{i-0fe7a5031fb82cc5b 0297efe3-5070-45f1-8aa2-9a1056b17b59 29165 0 2022-11-23 16:04:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0fe7a5031fb82cc5b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0fe7a5031fb82cc5b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.89 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-930":"i-0fe7a5031fb82cc5b","ebs.csi.aws.com":"i-0fe7a5031fb82cc5b"} io.cilium.network.ipv4-cilium-host:172.20.57.57 io.cilium.network.ipv4-health-ip:172.20.59.99 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-11-23 16:04:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:03 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:21:48 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 16:21:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0fe7a5031fb82cc5b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:03 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:21:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:21:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:54 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.89,},NodeAddress{Type:ExternalIP,Address:34.221.99.167,},NodeAddress{Type:InternalDNS,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-99-167.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29292cb2c053c780af3b39ea2fa8c6,SystemUUID:ec29292c-b2c0-53c7-80af-3b39ea2fa8c6,BootID:ff4df73e-ccb2-4109-99ca-04a1b7ab5611,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-930^ed65e6a3-6b4a-11ed-9ffc-1aa129783697],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-930^ed65e6a3-6b4a-11ed-9ffc-1aa129783697,DevicePath:,},},Config:nil,},}
Nov 23 16:22:02.333: INFO:
Logging kubelet events for node i-0fe7a5031fb82cc5b
Nov 23 16:22:02.406: INFO:
Logging pods the kubelet thinks is on node i-0fe7a5031fb82cc5b
Nov 23 16:22:02.478: INFO: ebs-csi-node-8ffwl started at 2022-11-23 16:04:34 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:22:02.478: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:22:02.478: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:22:02.478: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:22:02.478: INFO: cilium-727pv started at 2022-11-23 16:04:34 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:22:02.478: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:22:02.478: INFO: Container cilium-agent ready: true, restart count 0
Nov 23 16:22:02.478: INFO: coredns-6c7bddbb75-bpdc6 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:02.478: INFO: Container coredns ready: true, restart count 0
Nov 23 16:22:02.478: INFO: coredns-autoscaler-db7b97744-r4jm7 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:02.478: INFO: Container autoscaler ready: true, restart count 0
Nov 23 16:22:02.478: INFO: inline-volume-tester-dvvp2 started at 2022-11-23 16:21:48 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:02.478: INFO: Container csi-volume-tester ready: true, restart count 0
Nov 23 16:22:02.478: INFO: ebs-csi-controller-847c4b8bc4-lxdkr started at 2022-11-23 16:04:54 +0000 UTC (0+5 container statuses recorded)
Nov 23 16:22:02.478: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:22:02.478: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:22:02.478: INFO: Container csi-resizer ready: true, restart count 0
Nov 23 16:22:02.478: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:22:02.478: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:22:02.478: INFO: netserver-3 started at 2022-11-23 16:21:30 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:22:02.478: INFO: Container webserver ready: true, restart count 0
Nov 23 16:22:02.478: INFO: csi-hostpathplugin-0 started at 2022-11-23 16:21:44 +0000 UTC (0+7 container statuses recorded)
Nov 23 16:22:02.478: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:22:02.478: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:22:02.478: INFO: Container csi-resizer ready: true, restart count 0
Nov 23 16:22:02.478: INFO: Container csi-snapshotter ready: true, restart count 0
Nov 23 16:22:02.478: INFO: Container hostpath ready: true, restart count 0
Nov 23 16:22:02.478: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:22:02.478: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:22:02.902: INFO:
Latency metrics for node i-0fe7a5031fb82cc5b
Nov 23 16:22:02.902: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
�[1mSTEP:�[0m Destroying namespace "fsgroupchangepolicy-801" for this suite. �[38;5;243m11/23/22 16:22:02.967�[0m
from junit_01.xml
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(Always\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\svolume\scontents\sownership\schanged\svia\schgrp\sin\sfirst\spod\,\snew\spod\swith\sdifferent\sfsgroup\sapplied\sto\sthe\svolume\scontents$'
test/e2e/storage/testsuites/fsgroupchangepolicy.go:264
k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc00146e2c0, 0xc0018fde30, 0x1, {0x0, 0x0}, {0x0, 0x0})
test/e2e/storage/testsuites/fsgroupchangepolicy.go:264 +0x151
k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3()
test/e2e/storage/testsuites/fsgroupchangepolicy.go:235 +0x298
{"msg":"FAILED External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents","completed":8,"skipped":63,"failed":1,"failures":["External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents"]}
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
test/e2e/framework/framework.go:186
�[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m11/23/22 16:09:02.546�[0m
Nov 23 16:09:02.547: INFO: >>> kubeConfig: /root/.kube/config
�[1mSTEP:�[0m Building a namespace api object, basename fsgroupchangepolicy �[38;5;243m11/23/22 16:09:02.547�[0m
�[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m11/23/22 16:09:02.742�[0m
�[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m11/23/22 16:09:02.871�[0m
[It] (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents
test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
Nov 23 16:09:03.000: INFO: Creating resource for dynamic PV
Nov 23 16:09:03.000: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi}
�[1mSTEP:�[0m creating a StorageClass fsgroupchangepolicy-2350-e2e-sc95rsw �[38;5;243m11/23/22 16:09:03.001�[0m
�[1mSTEP:�[0m creating a claim �[38;5;243m11/23/22 16:09:03.068�[0m
Nov 23 16:09:03.068: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
�[1mSTEP:�[0m Creating Pod in namespace fsgroupchangepolicy-2350 with fsgroup 1000 �[38;5;243m11/23/22 16:09:03.206�[0m
Nov 23 16:09:03.272: INFO: Waiting up to 5m0s for pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972" in namespace "fsgroupchangepolicy-2350" to be "running"
Nov 23 16:09:03.337: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 64.501618ms
Nov 23 16:09:05.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2.129212988s
Nov 23 16:09:07.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4.129937921s
Nov 23 16:09:09.403: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 6.131151751s
Nov 23 16:09:11.407: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 8.134464585s
Nov 23 16:09:13.405: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 10.132442402s
Nov 23 16:09:15.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 12.129200487s
Nov 23 16:09:17.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 14.129473077s
Nov 23 16:09:19.404: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 16.131489087s
Nov 23 16:09:21.403: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 18.131227806s
Nov 23 16:09:23.407: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 20.134993275s
Nov 23 16:09:25.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 22.128758058s
Nov 23 16:09:27.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 24.129220972s
Nov 23 16:09:29.406: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 26.133866861s
Nov 23 16:09:31.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 28.129987302s
Nov 23 16:09:33.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 30.130156787s
Nov 23 16:09:35.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 32.129652801s
Nov 23 16:09:37.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 34.12956109s
Nov 23 16:09:39.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 36.129262804s
Nov 23 16:09:41.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 38.129128967s
Nov 23 16:09:43.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 40.129556076s
Nov 23 16:09:45.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 42.129172497s
Nov 23 16:09:47.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 44.130130629s
Nov 23 16:09:49.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 46.130039531s
Nov 23 16:09:51.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 48.129769614s
Nov 23 16:09:53.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 50.129669748s
Nov 23 16:09:55.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 52.129684939s
Nov 23 16:09:57.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 54.129623255s
Nov 23 16:09:59.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 56.129725157s
Nov 23 16:10:01.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 58.129508939s
Nov 23 16:10:03.404: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.131639532s
Nov 23 16:10:05.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.129682194s
Nov 23 16:10:07.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.129926794s
Nov 23 16:10:09.403: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.130545324s
Nov 23 16:10:11.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.129499424s
Nov 23 16:10:13.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.129907398s
Nov 23 16:10:15.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.130039852s
Nov 23 16:10:17.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.129378973s
Nov 23 16:10:19.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.129885381s
Nov 23 16:10:21.403: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.130872871s
Nov 23 16:10:23.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.129209198s
Nov 23 16:10:25.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.129481794s
Nov 23 16:10:27.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.128840461s
Nov 23 16:10:29.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.129749195s
Nov 23 16:10:31.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.129091508s
Nov 23 16:10:33.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.129606709s
Nov 23 16:10:35.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.129741878s
Nov 23 16:10:37.403: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.130770195s
Nov 23 16:10:39.405: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.13283411s
Nov 23 16:10:41.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m38.128161085s
Nov 23 16:10:43.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m40.128132912s
Nov 23 16:10:45.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m42.128570966s
Nov 23 16:10:47.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.128714554s
Nov 23 16:10:49.403: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.130621833s
Nov 23 16:10:51.404: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.132330457s
Nov 23 16:10:53.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m50.13023674s
Nov 23 16:10:55.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m52.129754188s
Nov 23 16:10:57.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m54.129096353s
Nov 23 16:10:59.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m56.129043562s
Nov 23 16:11:01.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 1m58.128259221s
Nov 23 16:11:03.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m0.128066149s
Nov 23 16:11:05.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m2.128741835s
Nov 23 16:11:07.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m4.128165582s
Nov 23 16:11:09.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.128635024s
Nov 23 16:11:11.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.128177106s
Nov 23 16:11:13.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.128625164s
Nov 23 16:11:15.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.127749305s
Nov 23 16:11:17.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.127847871s
Nov 23 16:11:19.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.129162613s
Nov 23 16:11:21.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.128315896s
Nov 23 16:11:23.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.128416113s
Nov 23 16:11:25.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.128393699s
Nov 23 16:11:27.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.12877687s
Nov 23 16:11:29.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.129272281s
Nov 23 16:11:31.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.128887792s
Nov 23 16:11:33.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.128024799s
Nov 23 16:11:35.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.12792564s
Nov 23 16:11:37.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.128468013s
Nov 23 16:11:39.405: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.133220614s
Nov 23 16:11:41.405: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.132814644s
Nov 23 16:11:43.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.128474807s
Nov 23 16:11:45.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.128080474s
Nov 23 16:11:47.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.129439494s
Nov 23 16:11:49.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.128227286s
Nov 23 16:11:51.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.128476262s
Nov 23 16:11:53.436: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.164303069s
Nov 23 16:11:55.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.129131296s
Nov 23 16:11:57.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.128291789s
Nov 23 16:11:59.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.128787213s
Nov 23 16:12:01.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 2m58.129630794s
Nov 23 16:12:03.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m0.128974251s
Nov 23 16:12:05.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m2.128538759s
Nov 23 16:12:07.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.128744308s
Nov 23 16:12:09.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m6.128778042s
Nov 23 16:12:11.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m8.128335075s
Nov 23 16:12:13.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.128205507s
Nov 23 16:12:15.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.129287898s
Nov 23 16:12:17.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m14.128976416s
Nov 23 16:12:19.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m16.128151209s
Nov 23 16:12:21.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m18.129166651s
Nov 23 16:12:23.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m20.128443154s
Nov 23 16:12:25.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m22.128568955s
Nov 23 16:12:27.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.128347823s
Nov 23 16:12:29.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.128310222s
Nov 23 16:12:31.405: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.133342659s
Nov 23 16:12:33.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.128231941s
Nov 23 16:12:35.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.129918011s
Nov 23 16:12:37.403: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m34.130678858s
Nov 23 16:12:39.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m36.129525275s
Nov 23 16:12:41.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m38.129820434s
Nov 23 16:12:43.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m40.128612652s
Nov 23 16:12:45.403: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m42.130555724s
Nov 23 16:12:47.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m44.128287208s
Nov 23 16:12:49.408: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m46.136329429s
Nov 23 16:12:51.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m48.129520896s
Nov 23 16:12:53.407: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m50.134803701s
Nov 23 16:12:55.403: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m52.131291s
Nov 23 16:12:57.400: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m54.12794386s
Nov 23 16:12:59.408: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m56.135474594s
Nov 23 16:13:01.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 3m58.128782392s
Nov 23 16:13:03.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m0.129330534s
Nov 23 16:13:05.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m2.129396474s
Nov 23 16:13:07.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m4.129425071s
Nov 23 16:13:09.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.129159941s
Nov 23 16:13:11.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.129941874s
Nov 23 16:13:13.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.129966956s
Nov 23 16:13:15.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.130287798s
Nov 23 16:13:17.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.129381729s
Nov 23 16:13:19.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m16.129927202s
Nov 23 16:13:21.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m18.129582102s
Nov 23 16:13:23.403: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m20.131083318s
Nov 23 16:13:25.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m22.129841055s
Nov 23 16:13:27.403: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m24.130972794s
Nov 23 16:13:29.403: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.130911373s
Nov 23 16:13:31.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.129928702s
Nov 23 16:13:33.404: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.132296533s
Nov 23 16:13:35.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.130217077s
Nov 23 16:13:37.403: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.131199961s
Nov 23 16:13:39.403: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.130636227s
Nov 23 16:13:41.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.129669948s
Nov 23 16:13:43.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.129874281s
Nov 23 16:13:45.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.129562293s
Nov 23 16:13:47.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.130111574s
Nov 23 16:13:49.403: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.130442196s
Nov 23 16:13:51.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.129852776s
Nov 23 16:13:53.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.1294243s
Nov 23 16:13:55.401: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.129001771s
Nov 23 16:13:57.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.129380112s
Nov 23 16:13:59.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.129456206s
Nov 23 16:14:01.402: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.129414615s
Nov 23 16:14:03.404: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.131579403s
Nov 23 16:14:03.468: INFO: Pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.196328784s
Nov 23 16:14:03.469: INFO: Unexpected error:
<*errors.errorString | 0xc000e9cef0>: {
s: "pod \"pod-e79dc464-7b66-4d53-8552-bae17bbd6972\" is not Running: timed out while waiting for pod fsgroupchangepolicy-2350/pod-e79dc464-7b66-4d53-8552-bae17bbd6972 to be running",
}
Nov 23 16:14:03.469: FAIL: pod "pod-e79dc464-7b66-4d53-8552-bae17bbd6972" is not Running: timed out while waiting for pod fsgroupchangepolicy-2350/pod-e79dc464-7b66-4d53-8552-bae17bbd6972 to be running
Full Stack Trace
k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc00146e2c0, 0xc0018fde30, 0x1, {0x0, 0x0}, {0x0, 0x0})
test/e2e/storage/testsuites/fsgroupchangepolicy.go:264 +0x151
k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3()
test/e2e/storage/testsuites/fsgroupchangepolicy.go:235 +0x298
�[1mSTEP:�[0m Deleting pvc �[38;5;243m11/23/22 16:14:03.469�[0m
Nov 23 16:14:03.537: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comdpnt2"
�[1mSTEP:�[0m Deleting sc �[38;5;243m11/23/22 16:14:03.606�[0m
[AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
test/e2e/framework/framework.go:187
�[1mSTEP:�[0m Collecting events from namespace "fsgroupchangepolicy-2350". �[38;5;243m11/23/22 16:14:03.674�[0m
�[1mSTEP:�[0m Found 5 events. �[38;5;243m11/23/22 16:14:03.738�[0m
Nov 23 16:14:03.738: INFO: At 2022-11-23 16:09:03 +0000 UTC - event for ebs.csi.aws.comdpnt2: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
Nov 23 16:14:03.738: INFO: At 2022-11-23 16:09:03 +0000 UTC - event for ebs.csi.aws.comdpnt2: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
Nov 23 16:14:03.738: INFO: At 2022-11-23 16:09:03 +0000 UTC - event for ebs.csi.aws.comdpnt2: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-2350/ebs.csi.aws.comdpnt2"
Nov 23 16:14:03.738: INFO: At 2022-11-23 16:09:13 +0000 UTC - event for ebs.csi.aws.comdpnt2: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-2350-e2e-sc95rsw": rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 23 16:14:03.738: INFO: At 2022-11-23 16:10:34 +0000 UTC - event for ebs.csi.aws.comdpnt2: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-2350-e2e-sc95rsw": rpc error: code = Internal desc = Could not create volume "pvc-9f3a4820-2162-422d-801c-44900b5cd9d8": could not create volume in EC2: RequestCanceled: request context canceled
caused by: context deadline exceeded
Nov 23 16:14:03.802: INFO: POD NODE PHASE GRACE CONDITIONS
Nov 23 16:14:03.802: INFO: pod-e79dc464-7b66-4d53-8552-bae17bbd6972 Pending []
Nov 23 16:14:03.802: INFO:
Nov 23 16:14:03.932: INFO:
Logging node info for node i-060d86c6ffc5622b0
Nov 23 16:14:03.999: INFO: Node Info: &Node{ObjectMeta:{i-060d86c6ffc5622b0 b49ff0b3-8686-4c0f-b991-1d427ca43e2a 7999 0 2022-11-23 16:02:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-060d86c6ffc5622b0 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.40.57 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-060d86c6ffc5622b0"} io.cilium.network.ipv4-cilium-host:172.20.57.204 io.cilium.network.ipv4-health-ip:172.20.37.200 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 16:02:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2022-11-23 16:03:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:03:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:10:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-060d86c6ffc5622b0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:04:42 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:10:12 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:10:12 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:10:12 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:10:12 +0000 UTC,LastTransitionTime:2022-11-23 16:03:24 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.57,},NodeAddress{Type:ExternalIP,Address:54.200.161.23,},NodeAddress{Type:InternalDNS,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-200-161-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec226e0d77c347467223a292b33eb0c5,SystemUUID:ec226e0d-77c3-4746-7223-a292b33eb0c5,BootID:936b9dd2-17e7-49ac-8c1e-7556bf2c9ee8,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.25.4],SizeBytes:129085904,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.25.4],SizeBytes:118436153,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.25.4],SizeBytes:51921208,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42928334,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41477333,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:85d3f1e9dacc72531445989bb10999e1e70ebc409d11be57e5baa5f031a893b0 registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1],SizeBytes:18257577,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955428,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov 23 16:14:04.000: INFO:
Logging kubelet events for node i-060d86c6ffc5622b0
Nov 23 16:14:04.067: INFO:
Logging pods the kubelet thinks is on node i-060d86c6ffc5622b0
Nov 23 16:14:04.139: INFO: etcd-manager-events-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.139: INFO: Container etcd-manager ready: true, restart count 0
Nov 23 16:14:04.139: INFO: kops-controller-7w98n started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.139: INFO: Container kops-controller ready: true, restart count 0
Nov 23 16:14:04.139: INFO: aws-cloud-controller-manager-dzzkp started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.139: INFO: Container aws-cloud-controller-manager ready: true, restart count 0
Nov 23 16:14:04.139: INFO: etcd-manager-main-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.139: INFO: Container etcd-manager ready: true, restart count 0
Nov 23 16:14:04.139: INFO: kube-apiserver-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+2 container statuses recorded)
Nov 23 16:14:04.139: INFO: Container healthcheck ready: true, restart count 0
Nov 23 16:14:04.139: INFO: Container kube-apiserver ready: true, restart count 1
Nov 23 16:14:04.139: INFO: kube-controller-manager-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.139: INFO: Container kube-controller-manager ready: true, restart count 2
Nov 23 16:14:04.139: INFO: kube-scheduler-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.139: INFO: Container kube-scheduler ready: true, restart count 0
Nov 23 16:14:04.139: INFO: dns-controller-89c6d5f67-9hm78 started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.139: INFO: Container dns-controller ready: true, restart count 0
Nov 23 16:14:04.139: INFO: cilium-f2gg7 started at 2022-11-23 16:03:06 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:14:04.139: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:14:04.139: INFO: Container cilium-agent ready: true, restart count 1
Nov 23 16:14:04.139: INFO: cilium-operator-7f7bcdf787-xnz2q started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.139: INFO: Container cilium-operator ready: true, restart count 1
Nov 23 16:14:04.139: INFO: ebs-csi-node-vpv4t started at 2022-11-23 16:03:06 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:14:04.139: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:14:04.139: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:14:04.139: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:14:04.399: INFO:
Latency metrics for node i-060d86c6ffc5622b0
Nov 23 16:14:04.399: INFO:
Logging node info for node i-0765e732a9635923b
Nov 23 16:14:04.467: INFO: Node Info: &Node{ObjectMeta:{i-0765e732a9635923b 10981661-cc27-412a-9e66-7a37930b898f 15572 0 2022-11-23 16:04:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0765e732a9635923b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0765e732a9635923b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.58.74 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0765e732a9635923b"} io.cilium.network.ipv4-cilium-host:172.20.54.198 io.cilium.network.ipv4-health-ip:172.20.60.24 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:07 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:13:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0765e732a9635923b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:07 +0000 UTC,LastTransitionTime:2022-11-23 16:05:07 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:11:38 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:11:38 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:11:38 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:11:38 +0000 UTC,LastTransitionTime:2022-11-23 16:05:00 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.58.74,},NodeAddress{Type:ExternalIP,Address:52.25.125.23,},NodeAddress{Type:InternalDNS,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-25-125-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec23d129b4036d8d979053444dfae008,SystemUUID:ec23d129-b403-6d8d-9790-53444dfae008,BootID:fe390540-4601-456e-9686-93a36ea3dc8f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov 23 16:14:04.467: INFO:
Logging kubelet events for node i-0765e732a9635923b
Nov 23 16:14:04.535: INFO:
Logging pods the kubelet thinks is on node i-0765e732a9635923b
Nov 23 16:14:04.610: INFO: coredns-6c7bddbb75-2n2dw started at 2022-11-23 16:05:19 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.610: INFO: Container coredns ready: true, restart count 0
Nov 23 16:14:04.610: INFO: fail-once-local-w7nbw started at 2022-11-23 16:13:48 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.610: INFO: Container c ready: false, restart count 1
Nov 23 16:14:04.610: INFO: ebs-csi-node-5khlc started at 2022-11-23 16:04:40 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:14:04.610: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:14:04.610: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:14:04.610: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:14:04.610: INFO: affinity-nodeport-krcbx started at 2022-11-23 16:14:03 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.610: INFO: Container affinity-nodeport ready: false, restart count 0
Nov 23 16:14:04.610: INFO: forbid-27820334-7wz8s started at 2022-11-23 16:14:00 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.610: INFO: Container c ready: true, restart count 0
Nov 23 16:14:04.610: INFO: pod-projected-configmaps-cb3a9864-bc2f-44c7-92b9-e72d4d58769d started at 2022-11-23 16:13:10 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:14:04.610: INFO: Container createcm-volume-test ready: true, restart count 0
Nov 23 16:14:04.610: INFO: Container delcm-volume-test ready: true, restart count 0
Nov 23 16:14:04.610: INFO: Container updcm-volume-test ready: true, restart count 0
Nov 23 16:14:04.610: INFO: netserver-0 started at 2022-11-23 16:13:59 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.610: INFO: Container webserver ready: false, restart count 0
Nov 23 16:14:04.610: INFO: cilium-czwdq started at 2022-11-23 16:04:40 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:14:04.610: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:14:04.610: INFO: Container cilium-agent ready: true, restart count 0
Nov 23 16:14:04.610: INFO: netserver-0 started at 2022-11-23 16:13:45 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.610: INFO: Container webserver ready: false, restart count 0
Nov 23 16:14:04.610: INFO: fail-once-local-4k2nf started at 2022-11-23 16:13:48 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.610: INFO: Container c ready: false, restart count 1
Nov 23 16:14:04.610: INFO: fail-once-local-df6f2 started at 2022-11-23 16:13:55 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.610: INFO: Container c ready: false, restart count 1
Nov 23 16:14:04.610: INFO: fail-once-local-sfr9n started at 2022-11-23 16:13:55 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:04.610: INFO: Container c ready: false, restart count 1
Nov 23 16:14:04.936: INFO:
Latency metrics for node i-0765e732a9635923b
Nov 23 16:14:04.937: INFO:
Logging node info for node i-0ade3c3082581cb52
Nov 23 16:14:05.002: INFO: Node Info: &Node{ObjectMeta:{i-0ade3c3082581cb52 2305662a-7d05-4979-bcec-220b8b4672db 13497 0 2022-11-23 16:04:42 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0ade3c3082581cb52 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0ade3c3082581cb52 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.56.12 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0ade3c3082581cb52"} io.cilium.network.ipv4-cilium-host:172.20.45.237 io.cilium.network.ipv4-health-ip:172.20.55.90 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:12:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0ade3c3082581cb52,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:12 +0000 UTC,LastTransitionTime:2022-11-23 16:05:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:12:21 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:12:21 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:12:21 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:12:21 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.56.12,},NodeAddress{Type:ExternalIP,Address:35.92.243.169,},NodeAddress{Type:InternalDNS,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-92-243-169.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bd980dfb52643cb4857d57b45824b,SystemUUID:ec2bd980-dfb5-2643-cb48-57d57b45824b,BootID:55912341-dce8-4199-a57c-c8703e24af46,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov 23 16:14:05.002: INFO:
Logging kubelet events for node i-0ade3c3082581cb52
Nov 23 16:14:05.069: INFO:
Logging pods the kubelet thinks is on node i-0ade3c3082581cb52
Nov 23 16:14:05.142: INFO: pod-ephm-test-configmap-r7fl started at 2022-11-23 16:13:45 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:05.142: INFO: Container test-container-subpath-configmap-r7fl ready: false, restart count 0
Nov 23 16:14:05.142: INFO: test-ss-0 started at 2022-11-23 16:13:25 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:05.142: INFO: Container webserver ready: true, restart count 0
Nov 23 16:14:05.142: INFO: pod-ready started at 2022-11-23 16:13:28 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:05.142: INFO: Container pod-readiness-gate ready: true, restart count 0
Nov 23 16:14:05.142: INFO: netserver-1 started at 2022-11-23 16:13:59 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:05.142: INFO: Container webserver ready: false, restart count 0
Nov 23 16:14:05.142: INFO: liveness-38645321-d2f1-4818-b657-547947353222 started at 2022-11-23 16:12:51 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:05.142: INFO: Container agnhost-container ready: true, restart count 3
Nov 23 16:14:05.142: INFO: csi-mockplugin-0 started at 2022-11-23 16:14:04 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:14:05.142: INFO: Container csi-provisioner ready: false, restart count 0
Nov 23 16:14:05.142: INFO: Container driver-registrar ready: false, restart count 0
Nov 23 16:14:05.142: INFO: Container mock ready: false, restart count 0
Nov 23 16:14:05.142: INFO: cilium-q5tsk started at 2022-11-23 16:04:43 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:14:05.142: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:14:05.142: INFO: Container cilium-agent ready: true, restart count 0
Nov 23 16:14:05.142: INFO: pod-qos-class-ef170dd3-fd69-418f-ac17-4fb3aedb39a6 started at 2022-11-23 16:13:39 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:05.142: INFO: Container agnhost ready: false, restart count 0
Nov 23 16:14:05.142: INFO: ebs-csi-node-t56sd started at 2022-11-23 16:04:43 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:14:05.142: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:14:05.142: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:14:05.142: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:14:05.142: INFO: affinity-nodeport-sq79s started at 2022-11-23 16:14:03 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:05.142: INFO: Container affinity-nodeport ready: false, restart count 0
Nov 23 16:14:05.142: INFO: netserver-1 started at 2022-11-23 16:13:45 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:05.142: INFO: Container webserver ready: false, restart count 0
Nov 23 16:14:05.142: INFO: csi-mockplugin-resizer-0 started at 2022-11-23 16:14:04 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:05.142: INFO: Container csi-resizer ready: false, restart count 0
Nov 23 16:14:05.546: INFO:
Latency metrics for node i-0ade3c3082581cb52
Nov 23 16:14:05.546: INFO:
Logging node info for node i-0bf2756d84bae9168
Nov 23 16:14:05.611: INFO: Node Info: &Node{ObjectMeta:{i-0bf2756d84bae9168 2f26f127-5eb0-44f3-97f0-990045e7b900 15437 0 2022-11-23 16:04:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0bf2756d84bae9168 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0bf2756d84bae9168 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.218 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-323":"i-0bf2756d84bae9168","csi-hostpath-ephemeral-4263":"i-0bf2756d84bae9168","ebs.csi.aws.com":"i-0bf2756d84bae9168"} io.cilium.network.ipv4-cilium-host:172.20.35.201 io.cilium.network.ipv4-health-ip:172.20.51.170 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:02 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:13:35 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 16:13:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0bf2756d84bae9168,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:02 +0000 UTC,LastTransitionTime:2022-11-23 16:05:02 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:13:35 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:13:35 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:13:35 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:13:35 +0000 UTC,LastTransitionTime:2022-11-23 16:04:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.218,},NodeAddress{Type:ExternalIP,Address:34.221.37.130,},NodeAddress{Type:InternalDNS,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-37-130.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d7db7f0ff759b7a6209e832f1dcea,SystemUUID:ec2d7db7-f0ff-759b-7a62-09e832f1dcea,BootID:8f294a85-a1ed-4dfb-bdb9-f8818e1f067f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f registry.k8s.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-323^a3a2194b-6b49-11ed-8275-0ef81ef9666a kubernetes.io/csi/csi-hostpath-ephemeral-4263^a3a6d07a-6b49-11ed-be66-1607fe83facc],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-4263^a3a6d07a-6b49-11ed-be66-1607fe83facc,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-323^a3a2194b-6b49-11ed-8275-0ef81ef9666a,DevicePath:,},},Config:nil,},}
Nov 23 16:14:05.612: INFO:
Logging kubelet events for node i-0bf2756d84bae9168
Nov 23 16:14:05.684: INFO:
Logging pods the kubelet thinks is on node i-0bf2756d84bae9168
Nov 23 16:14:05.759: INFO: hostexec-i-0bf2756d84bae9168-gn6f6 started at 2022-11-23 16:13:46 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:05.759: INFO: Container agnhost-container ready: true, restart count 0
Nov 23 16:14:05.759: INFO: ebs-csi-node-crshz started at 2022-11-23 16:04:35 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:14:05.759: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:14:05.759: INFO: netserver-2 started at 2022-11-23 16:13:45 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:05.759: INFO: Container webserver ready: true, restart count 0
Nov 23 16:14:05.759: INFO: csi-hostpathplugin-0 started at 2022-11-23 16:12:31 +0000 UTC (0+7 container statuses recorded)
Nov 23 16:14:05.759: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container csi-resizer ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container csi-snapshotter ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container hostpath ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:14:05.759: INFO: netserver-2 started at 2022-11-23 16:13:59 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:05.759: INFO: Container webserver ready: false, restart count 0
Nov 23 16:14:05.759: INFO: csi-hostpathplugin-0 started at 2022-11-23 16:12:31 +0000 UTC (0+7 container statuses recorded)
Nov 23 16:14:05.759: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container csi-resizer ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container csi-snapshotter ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container hostpath ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:14:05.759: INFO: cilium-scm24 started at 2022-11-23 16:04:35 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:14:05.759: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container cilium-agent ready: true, restart count 0
Nov 23 16:14:05.759: INFO: ebs-csi-controller-847c4b8bc4-kwpsb started at 2022-11-23 16:04:56 +0000 UTC (0+5 container statuses recorded)
Nov 23 16:14:05.759: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container csi-resizer ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:14:05.759: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:14:06.133: INFO:
Latency metrics for node i-0bf2756d84bae9168
Nov 23 16:14:06.133: INFO:
Logging node info for node i-0fe7a5031fb82cc5b
Nov 23 16:14:06.198: INFO: Node Info: &Node{ObjectMeta:{i-0fe7a5031fb82cc5b 0297efe3-5070-45f1-8aa2-9a1056b17b59 15972 0 2022-11-23 16:04:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0fe7a5031fb82cc5b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0fe7a5031fb82cc5b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.89 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-9785":"i-0fe7a5031fb82cc5b","ebs.csi.aws.com":"i-0fe7a5031fb82cc5b"} io.cilium.network.ipv4-cilium-host:172.20.57.57 io.cilium.network.ipv4-health-ip:172.20.59.99 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-11-23 16:04:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:03 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:13:49 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 16:13:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0fe7a5031fb82cc5b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:03 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:13:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:13:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:13:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:13:55 +0000 UTC,LastTransitionTime:2022-11-23 16:04:54 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.89,},NodeAddress{Type:ExternalIP,Address:34.221.99.167,},NodeAddress{Type:InternalDNS,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-99-167.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29292cb2c053c780af3b39ea2fa8c6,SystemUUID:ec29292c-b2c0-53c7-80af-3b39ea2fa8c6,BootID:ff4df73e-ccb2-4109-99ca-04a1b7ab5611,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-9785^cf4b8e61-6b49-11ed-b309-668c69ba5f59],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-9785^cf4b8e61-6b49-11ed-b309-668c69ba5f59,DevicePath:,},},Config:nil,},}
Nov 23 16:14:06.198: INFO:
Logging kubelet events for node i-0fe7a5031fb82cc5b
Nov 23 16:14:06.265: INFO:
Logging pods the kubelet thinks is on node i-0fe7a5031fb82cc5b
Nov 23 16:14:06.358: INFO: netserver-3 started at 2022-11-23 16:13:45 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:06.358: INFO: Container webserver ready: true, restart count 0
Nov 23 16:14:06.358: INFO: pod-f7a05b8d-3f4e-4a23-a441-eb9866a6484b started at 2022-11-23 16:13:48 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:06.358: INFO: Container write-pod ready: true, restart count 0
Nov 23 16:14:06.358: INFO: ebs-csi-node-8ffwl started at 2022-11-23 16:04:34 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:14:06.358: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:14:06.358: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:14:06.358: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:14:06.358: INFO: coredns-autoscaler-db7b97744-r4jm7 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:06.358: INFO: Container autoscaler ready: true, restart count 0
Nov 23 16:14:06.358: INFO: csi-hostpathplugin-0 started at 2022-11-23 16:13:44 +0000 UTC (0+7 container statuses recorded)
Nov 23 16:14:06.358: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:14:06.358: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:14:06.358: INFO: Container csi-resizer ready: true, restart count 0
Nov 23 16:14:06.358: INFO: Container csi-snapshotter ready: true, restart count 0
Nov 23 16:14:06.358: INFO: Container hostpath ready: true, restart count 0
Nov 23 16:14:06.358: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:14:06.358: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:14:06.358: INFO: hostexec-i-0fe7a5031fb82cc5b-r288s started at 2022-11-23 16:13:49 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:06.358: INFO: Container agnhost-container ready: true, restart count 0
Nov 23 16:14:06.358: INFO: affinity-nodeport-clh7h started at 2022-11-23 16:14:03 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:06.358: INFO: Container affinity-nodeport ready: true, restart count 0
Nov 23 16:14:06.358: INFO: cilium-727pv started at 2022-11-23 16:04:34 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:14:06.358: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:14:06.358: INFO: Container cilium-agent ready: true, restart count 0
Nov 23 16:14:06.358: INFO: ebs-csi-controller-847c4b8bc4-lxdkr started at 2022-11-23 16:04:54 +0000 UTC (0+5 container statuses recorded)
Nov 23 16:14:06.358: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:14:06.358: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:14:06.358: INFO: Container csi-resizer ready: true, restart count 0
Nov 23 16:14:06.358: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:14:06.358: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:14:06.358: INFO: coredns-6c7bddbb75-bpdc6 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:06.358: INFO: Container coredns ready: true, restart count 0
Nov 23 16:14:06.358: INFO: netserver-3 started at 2022-11-23 16:13:59 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:14:06.358: INFO: Container webserver ready: false, restart count 0
Nov 23 16:14:06.701: INFO:
Latency metrics for node i-0fe7a5031fb82cc5b
Nov 23 16:14:06.701: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
�[1mSTEP:�[0m Destroying namespace "fsgroupchangepolicy-2350" for this suite. �[38;5;243m11/23/22 16:14:06.768�[0m
from junit_01.xml
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(Always\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\svolume\scontents\sownership\schanged\svia\schgrp\sin\sfirst\spod\,\snew\spod\swith\ssame\sfsgroup\sapplied\sto\sthe\svolume\scontents$'
test/e2e/storage/testsuites/fsgroupchangepolicy.go:264
k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc0013ef4a0, 0xc002dd9e30, 0x1, {0x0, 0x0}, {0x0, 0x0})
test/e2e/storage/testsuites/fsgroupchangepolicy.go:264 +0x151
k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3()
test/e2e/storage/testsuites/fsgroupchangepolicy.go:235 +0x298
{"msg":"FAILED External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents","completed":17,"skipped":170,"failed":1,"failures":["External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents"]}
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
test/e2e/framework/framework.go:186
�[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m11/23/22 16:14:20.975�[0m
Nov 23 16:14:20.976: INFO: >>> kubeConfig: /root/.kube/config
�[1mSTEP:�[0m Building a namespace api object, basename fsgroupchangepolicy �[38;5;243m11/23/22 16:14:20.977�[0m
�[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m11/23/22 16:14:21.169�[0m
�[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m11/23/22 16:14:21.3�[0m
[It] (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents
test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
Nov 23 16:14:21.427: INFO: Creating resource for dynamic PV
Nov 23 16:14:21.427: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi}
�[1mSTEP:�[0m creating a StorageClass fsgroupchangepolicy-4290-e2e-scqqk87 �[38;5;243m11/23/22 16:14:21.428�[0m
�[1mSTEP:�[0m creating a claim �[38;5;243m11/23/22 16:14:21.495�[0m
Nov 23 16:14:21.496: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
�[1mSTEP:�[0m Creating Pod in namespace fsgroupchangepolicy-4290 with fsgroup 1000 �[38;5;243m11/23/22 16:14:21.628�[0m
Nov 23 16:14:21.693: INFO: Waiting up to 5m0s for pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2" in namespace "fsgroupchangepolicy-4290" to be "running"
Nov 23 16:14:21.757: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 63.524625ms
Nov 23 16:14:23.825: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2.131515141s
Nov 23 16:14:25.832: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4.138750106s
Nov 23 16:14:27.827: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 6.134107559s
Nov 23 16:14:29.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 8.127231502s
Nov 23 16:14:31.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 10.127472756s
Nov 23 16:14:33.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 12.127833496s
Nov 23 16:14:35.830: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 14.13726141s
Nov 23 16:14:37.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 16.128087772s
Nov 23 16:14:39.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 18.127213061s
Nov 23 16:14:41.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 20.127823762s
Nov 23 16:14:43.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 22.127092826s
Nov 23 16:14:45.827: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 24.133667169s
Nov 23 16:14:47.823: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 26.130362963s
Nov 23 16:14:49.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 28.127590447s
Nov 23 16:14:51.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 30.128209639s
Nov 23 16:14:53.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 32.127576571s
Nov 23 16:14:55.830: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 34.13667031s
Nov 23 16:14:57.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 36.127481219s
Nov 23 16:14:59.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 38.128278964s
Nov 23 16:15:01.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 40.128748569s
Nov 23 16:15:03.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 42.127285675s
Nov 23 16:15:05.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 44.12715946s
Nov 23 16:15:07.829: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 46.135622162s
Nov 23 16:15:09.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 48.127462718s
Nov 23 16:15:11.823: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 50.1297347s
Nov 23 16:15:13.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 52.127220415s
Nov 23 16:15:15.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 54.127591383s
Nov 23 16:15:17.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 56.127288159s
Nov 23 16:15:19.831: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 58.138451254s
Nov 23 16:15:21.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.128353789s
Nov 23 16:15:23.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.128068706s
Nov 23 16:15:25.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.127781847s
Nov 23 16:15:27.824: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.130897326s
Nov 23 16:15:29.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.127879997s
Nov 23 16:15:31.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.126987487s
Nov 23 16:15:33.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.129133574s
Nov 23 16:15:35.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.127818639s
Nov 23 16:15:37.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.128705186s
Nov 23 16:15:39.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.127539236s
Nov 23 16:15:41.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.128619787s
Nov 23 16:15:43.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.128952206s
Nov 23 16:15:45.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.128575988s
Nov 23 16:15:47.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.129030345s
Nov 23 16:15:49.827: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.133568981s
Nov 23 16:15:51.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.127974786s
Nov 23 16:15:53.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.127203923s
Nov 23 16:15:55.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.12780158s
Nov 23 16:15:57.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.127310291s
Nov 23 16:15:59.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m38.12829215s
Nov 23 16:16:01.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m40.127932386s
Nov 23 16:16:03.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m42.127397751s
Nov 23 16:16:05.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.127582265s
Nov 23 16:16:07.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.128036703s
Nov 23 16:16:09.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.127382016s
Nov 23 16:16:11.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m50.128346366s
Nov 23 16:16:13.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m52.128336466s
Nov 23 16:16:15.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m54.127347735s
Nov 23 16:16:17.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m56.128321941s
Nov 23 16:16:19.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 1m58.127683622s
Nov 23 16:16:21.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m0.128086518s
Nov 23 16:16:23.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m2.127808138s
Nov 23 16:16:25.836: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m4.142800675s
Nov 23 16:16:27.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.127968921s
Nov 23 16:16:29.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.1280957s
Nov 23 16:16:31.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.128037729s
Nov 23 16:16:33.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.128741281s
Nov 23 16:16:35.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.12929449s
Nov 23 16:16:37.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.127709796s
Nov 23 16:16:39.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.12825193s
Nov 23 16:16:41.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.12760434s
Nov 23 16:16:43.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.127203007s
Nov 23 16:16:45.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.127716016s
Nov 23 16:16:47.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.127879654s
Nov 23 16:16:49.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.127811793s
Nov 23 16:16:51.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.127291739s
Nov 23 16:16:53.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.129396366s
Nov 23 16:16:55.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.128625156s
Nov 23 16:16:57.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.127548004s
Nov 23 16:16:59.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.127788947s
Nov 23 16:17:01.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.128039685s
Nov 23 16:17:03.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.127809597s
Nov 23 16:17:05.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.127137902s
Nov 23 16:17:07.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.12823102s
Nov 23 16:17:09.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.128773756s
Nov 23 16:17:11.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.127794067s
Nov 23 16:17:13.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.12749295s
Nov 23 16:17:15.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.127676394s
Nov 23 16:17:17.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.129410341s
Nov 23 16:17:19.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 2m58.128460964s
Nov 23 16:17:21.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m0.127927683s
Nov 23 16:17:23.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m2.12742888s
Nov 23 16:17:25.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.127891943s
Nov 23 16:17:27.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m6.128251928s
Nov 23 16:17:29.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m8.127936943s
Nov 23 16:17:31.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.127990188s
Nov 23 16:17:33.824: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.131279041s
Nov 23 16:17:35.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m14.127761825s
Nov 23 16:17:37.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m16.127306328s
Nov 23 16:17:39.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m18.127907675s
Nov 23 16:17:41.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m20.127825452s
Nov 23 16:17:43.823: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m22.130022073s
Nov 23 16:17:45.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.127302256s
Nov 23 16:17:47.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.128449916s
Nov 23 16:17:49.828: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.134494931s
Nov 23 16:17:51.823: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.129543504s
Nov 23 16:17:53.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.12839949s
Nov 23 16:17:55.829: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m34.135885382s
Nov 23 16:17:57.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m36.127199796s
Nov 23 16:17:59.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m38.127271064s
Nov 23 16:18:01.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m40.1278581s
Nov 23 16:18:03.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m42.127155285s
Nov 23 16:18:05.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m44.127333494s
Nov 23 16:18:07.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m46.127841545s
Nov 23 16:18:09.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m48.127278907s
Nov 23 16:18:11.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m50.128229503s
Nov 23 16:18:13.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m52.126983051s
Nov 23 16:18:15.824: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m54.130639171s
Nov 23 16:18:17.820: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m56.127336556s
Nov 23 16:18:19.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 3m58.128186364s
Nov 23 16:18:21.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m0.127875739s
Nov 23 16:18:23.824: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m2.131248621s
Nov 23 16:18:25.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m4.128540314s
Nov 23 16:18:27.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.129164584s
Nov 23 16:18:29.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.128426945s
Nov 23 16:18:31.859: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.165635298s
Nov 23 16:18:33.824: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.131320202s
Nov 23 16:18:35.826: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.133376062s
Nov 23 16:18:37.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m16.128859351s
Nov 23 16:18:39.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m18.128815884s
Nov 23 16:18:41.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m20.129171395s
Nov 23 16:18:43.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m22.12936355s
Nov 23 16:18:45.830: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m24.136526444s
Nov 23 16:18:47.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.129447335s
Nov 23 16:18:49.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.128455417s
Nov 23 16:18:51.824: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.130866992s
Nov 23 16:18:53.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.128708695s
Nov 23 16:18:55.821: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.128442421s
Nov 23 16:18:57.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.129351505s
Nov 23 16:18:59.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.128832969s
Nov 23 16:19:01.829: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.135686014s
Nov 23 16:19:03.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.128713651s
Nov 23 16:19:05.823: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.130184387s
Nov 23 16:19:07.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.128902253s
Nov 23 16:19:09.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.129259699s
Nov 23 16:19:11.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.128650835s
Nov 23 16:19:13.824: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.130682802s
Nov 23 16:19:15.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.12873926s
Nov 23 16:19:17.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.1288976s
Nov 23 16:19:19.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.129234574s
Nov 23 16:19:21.822: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.128749667s
Nov 23 16:19:21.893: INFO: Pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.200214238s
Nov 23 16:19:21.893: INFO: Unexpected error:
<*errors.errorString | 0xc000f6a550>: {
s: "pod \"pod-80bf715e-29e3-4f55-9553-57b793cbd8a2\" is not Running: timed out while waiting for pod fsgroupchangepolicy-4290/pod-80bf715e-29e3-4f55-9553-57b793cbd8a2 to be running",
}
Nov 23 16:19:21.893: FAIL: pod "pod-80bf715e-29e3-4f55-9553-57b793cbd8a2" is not Running: timed out while waiting for pod fsgroupchangepolicy-4290/pod-80bf715e-29e3-4f55-9553-57b793cbd8a2 to be running
Full Stack Trace
k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc0013ef4a0, 0xc002dd9e30, 0x1, {0x0, 0x0}, {0x0, 0x0})
test/e2e/storage/testsuites/fsgroupchangepolicy.go:264 +0x151
k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3()
test/e2e/storage/testsuites/fsgroupchangepolicy.go:235 +0x298
�[1mSTEP:�[0m Deleting pvc �[38;5;243m11/23/22 16:19:21.894�[0m
Nov 23 16:19:21.958: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.com4h46j"
�[1mSTEP:�[0m Deleting sc �[38;5;243m11/23/22 16:19:22.026�[0m
[AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
test/e2e/framework/framework.go:187
�[1mSTEP:�[0m Collecting events from namespace "fsgroupchangepolicy-4290". �[38;5;243m11/23/22 16:19:22.093�[0m
�[1mSTEP:�[0m Found 5 events. �[38;5;243m11/23/22 16:19:22.158�[0m
Nov 23 16:19:22.158: INFO: At 2022-11-23 16:14:21 +0000 UTC - event for ebs.csi.aws.com4h46j: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
Nov 23 16:19:22.158: INFO: At 2022-11-23 16:14:21 +0000 UTC - event for ebs.csi.aws.com4h46j: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
Nov 23 16:19:22.158: INFO: At 2022-11-23 16:14:21 +0000 UTC - event for ebs.csi.aws.com4h46j: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-4290/ebs.csi.aws.com4h46j"
Nov 23 16:19:22.158: INFO: At 2022-11-23 16:14:31 +0000 UTC - event for ebs.csi.aws.com4h46j: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-4290-e2e-scqqk87": rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 23 16:19:22.158: INFO: At 2022-11-23 16:14:42 +0000 UTC - event for ebs.csi.aws.com4h46j: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-4290-e2e-scqqk87": rpc error: code = Internal desc = Could not create volume "pvc-c32cc1a4-e531-47d5-9eba-425e45454cab": could not create volume in EC2: RequestCanceled: request context canceled
caused by: context deadline exceeded
Nov 23 16:19:22.223: INFO: POD NODE PHASE GRACE CONDITIONS
Nov 23 16:19:22.223: INFO: pod-80bf715e-29e3-4f55-9553-57b793cbd8a2 Pending []
Nov 23 16:19:22.223: INFO:
Nov 23 16:19:22.355: INFO:
Logging node info for node i-060d86c6ffc5622b0
Nov 23 16:19:22.420: INFO: Node Info: &Node{ObjectMeta:{i-060d86c6ffc5622b0 b49ff0b3-8686-4c0f-b991-1d427ca43e2a 18859 0 2022-11-23 16:02:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-060d86c6ffc5622b0 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.40.57 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-060d86c6ffc5622b0"} io.cilium.network.ipv4-cilium-host:172.20.57.204 io.cilium.network.ipv4-health-ip:172.20.37.200 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 16:02:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2022-11-23 16:03:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:03:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:15:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-060d86c6ffc5622b0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:04:42 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:15:18 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:15:18 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:15:18 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:15:18 +0000 UTC,LastTransitionTime:2022-11-23 16:03:24 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.57,},NodeAddress{Type:ExternalIP,Address:54.200.161.23,},NodeAddress{Type:InternalDNS,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-200-161-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec226e0d77c347467223a292b33eb0c5,SystemUUID:ec226e0d-77c3-4746-7223-a292b33eb0c5,BootID:936b9dd2-17e7-49ac-8c1e-7556bf2c9ee8,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.25.4],SizeBytes:129085904,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.25.4],SizeBytes:118436153,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.25.4],SizeBytes:51921208,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42928334,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41477333,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:85d3f1e9dacc72531445989bb10999e1e70ebc409d11be57e5baa5f031a893b0 registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1],SizeBytes:18257577,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955428,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov 23 16:19:22.421: INFO:
Logging kubelet events for node i-060d86c6ffc5622b0
Nov 23 16:19:22.489: INFO:
Logging pods the kubelet thinks is on node i-060d86c6ffc5622b0
Nov 23 16:19:22.560: INFO: etcd-manager-events-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:22.560: INFO: Container etcd-manager ready: true, restart count 0
Nov 23 16:19:22.560: INFO: kops-controller-7w98n started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:22.560: INFO: Container kops-controller ready: true, restart count 0
Nov 23 16:19:22.560: INFO: cilium-operator-7f7bcdf787-xnz2q started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:22.560: INFO: Container cilium-operator ready: true, restart count 1
Nov 23 16:19:22.560: INFO: aws-cloud-controller-manager-dzzkp started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:22.560: INFO: Container aws-cloud-controller-manager ready: true, restart count 0
Nov 23 16:19:22.560: INFO: etcd-manager-main-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:22.560: INFO: Container etcd-manager ready: true, restart count 0
Nov 23 16:19:22.560: INFO: kube-apiserver-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+2 container statuses recorded)
Nov 23 16:19:22.560: INFO: Container healthcheck ready: true, restart count 0
Nov 23 16:19:22.560: INFO: Container kube-apiserver ready: true, restart count 1
Nov 23 16:19:22.560: INFO: kube-controller-manager-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:22.560: INFO: Container kube-controller-manager ready: true, restart count 2
Nov 23 16:19:22.560: INFO: kube-scheduler-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:22.560: INFO: Container kube-scheduler ready: true, restart count 0
Nov 23 16:19:22.560: INFO: dns-controller-89c6d5f67-9hm78 started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:22.560: INFO: Container dns-controller ready: true, restart count 0
Nov 23 16:19:22.560: INFO: cilium-f2gg7 started at 2022-11-23 16:03:06 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:19:22.560: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:19:22.560: INFO: Container cilium-agent ready: true, restart count 1
Nov 23 16:19:22.560: INFO: ebs-csi-node-vpv4t started at 2022-11-23 16:03:06 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:19:22.560: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:19:22.560: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:19:22.560: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:19:22.820: INFO:
Latency metrics for node i-060d86c6ffc5622b0
Nov 23 16:19:22.820: INFO:
Logging node info for node i-0765e732a9635923b
Nov 23 16:19:22.885: INFO: Node Info: &Node{ObjectMeta:{i-0765e732a9635923b 10981661-cc27-412a-9e66-7a37930b898f 22789 0 2022-11-23 16:04:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0765e732a9635923b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0765e732a9635923b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.58.74 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-3492":"i-0765e732a9635923b","ebs.csi.aws.com":"i-0765e732a9635923b"} io.cilium.network.ipv4-cilium-host:172.20.54.198 io.cilium.network.ipv4-health-ip:172.20.60.24 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:07 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:17:43 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 16:17:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0765e732a9635923b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:07 +0000 UTC,LastTransitionTime:2022-11-23 16:05:07 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:17:44 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:17:44 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:17:44 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:17:44 +0000 UTC,LastTransitionTime:2022-11-23 16:05:00 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.58.74,},NodeAddress{Type:ExternalIP,Address:52.25.125.23,},NodeAddress{Type:InternalDNS,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-25-125-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec23d129b4036d8d979053444dfae008,SystemUUID:ec23d129-b403-6d8d-9790-53444dfae008,BootID:fe390540-4601-456e-9686-93a36ea3dc8f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-3492^5ac9536e-6b4a-11ed-8a0f-5aa1046b3a3a],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-3492^5ac9536e-6b4a-11ed-8a0f-5aa1046b3a3a,DevicePath:,},},Config:nil,},}
Nov 23 16:19:22.886: INFO:
Logging kubelet events for node i-0765e732a9635923b
Nov 23 16:19:22.953: INFO:
Logging pods the kubelet thinks is on node i-0765e732a9635923b
Nov 23 16:19:23.023: INFO: cilium-czwdq started at 2022-11-23 16:04:40 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:19:23.023: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:19:23.023: INFO: Container cilium-agent ready: true, restart count 0
Nov 23 16:19:23.023: INFO: coredns-6c7bddbb75-2n2dw started at 2022-11-23 16:05:19 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:23.023: INFO: Container coredns ready: true, restart count 0
Nov 23 16:19:23.023: INFO: inline-volume-tester-ktfx7 started at 2022-11-23 16:17:42 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:23.023: INFO: Container csi-volume-tester ready: true, restart count 0
Nov 23 16:19:23.023: INFO: ebs-csi-node-5khlc started at 2022-11-23 16:04:40 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:19:23.023: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:19:23.023: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:19:23.023: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:19:23.023: INFO: csi-hostpathplugin-0 started at 2022-11-23 16:17:39 +0000 UTC (0+7 container statuses recorded)
Nov 23 16:19:23.023: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:19:23.023: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:19:23.023: INFO: Container csi-resizer ready: true, restart count 0
Nov 23 16:19:23.023: INFO: Container csi-snapshotter ready: true, restart count 0
Nov 23 16:19:23.023: INFO: Container hostpath ready: true, restart count 0
Nov 23 16:19:23.023: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:19:23.023: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:19:23.400: INFO:
Latency metrics for node i-0765e732a9635923b
Nov 23 16:19:23.400: INFO:
Logging node info for node i-0ade3c3082581cb52
Nov 23 16:19:23.466: INFO: Node Info: &Node{ObjectMeta:{i-0ade3c3082581cb52 2305662a-7d05-4979-bcec-220b8b4672db 25723 0 2022-11-23 16:04:42 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0ade3c3082581cb52 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0ade3c3082581cb52 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.56.12 csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-1519":"i-0ade3c3082581cb52","ebs.csi.aws.com":"i-0ade3c3082581cb52"} io.cilium.network.ipv4-cilium-host:172.20.45.237 io.cilium.network.ipv4-health-ip:172.20.55.90 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:19:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0ade3c3082581cb52,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:12 +0000 UTC,LastTransitionTime:2022-11-23 16:05:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:19:18 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:19:18 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:19:18 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:19:18 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.56.12,},NodeAddress{Type:ExternalIP,Address:35.92.243.169,},NodeAddress{Type:InternalDNS,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ade3c3082581cb52.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-35-92-243-169.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bd980dfb52643cb4857d57b45824b,SystemUUID:ec2bd980-dfb5-2643-cb48-57d57b45824b,BootID:55912341-dce8-4199-a57c-c8703e24af46,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 registry.k8s.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov 23 16:19:23.466: INFO:
Logging kubelet events for node i-0ade3c3082581cb52
Nov 23 16:19:23.534: INFO:
Logging pods the kubelet thinks is on node i-0ade3c3082581cb52
Nov 23 16:19:23.603: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 16:18:49 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:23.603: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:19:23.603: INFO: csi-mockplugin-0 started at 2022-11-23 16:18:48 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:19:23.603: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:19:23.603: INFO: Container driver-registrar ready: true, restart count 0
Nov 23 16:19:23.603: INFO: Container mock ready: true, restart count 0
Nov 23 16:19:23.603: INFO: cilium-q5tsk started at 2022-11-23 16:04:43 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:19:23.603: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:19:23.603: INFO: Container cilium-agent ready: true, restart count 0
Nov 23 16:19:23.603: INFO: test-ss-0 started at 2022-11-23 16:19:18 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:23.603: INFO: Container webserver ready: true, restart count 0
Nov 23 16:19:23.603: INFO: pod-should-be-evictedbbf63ae3-fbea-413d-9153-679c13a5c0ee started at 2022-11-23 16:19:09 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:23.603: INFO: Container bar ready: true, restart count 0
Nov 23 16:19:23.603: INFO: ebs-csi-node-t56sd started at 2022-11-23 16:04:43 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:19:23.603: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:19:23.603: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:19:23.603: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:19:23.925: INFO:
Latency metrics for node i-0ade3c3082581cb52
Nov 23 16:19:23.925: INFO:
Logging node info for node i-0bf2756d84bae9168
Nov 23 16:19:23.991: INFO: Node Info: &Node{ObjectMeta:{i-0bf2756d84bae9168 2f26f127-5eb0-44f3-97f0-990045e7b900 25218 0 2022-11-23 16:04:34 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0bf2756d84bae9168 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0bf2756d84bae9168 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.54.218 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0bf2756d84bae9168"} io.cilium.network.ipv4-cilium-host:172.20.35.201 io.cilium.network.ipv4-health-ip:172.20.51.170 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:02 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:18:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0bf2756d84bae9168,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:02 +0000 UTC,LastTransitionTime:2022-11-23 16:05:02 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:16:08 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:16:08 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:16:08 +0000 UTC,LastTransitionTime:2022-11-23 16:04:34 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:16:08 +0000 UTC,LastTransitionTime:2022-11-23 16:04:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.218,},NodeAddress{Type:ExternalIP,Address:34.221.37.130,},NodeAddress{Type:InternalDNS,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0bf2756d84bae9168.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-37-130.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2d7db7f0ff759b7a6209e832f1dcea,SystemUUID:ec2d7db7-f0ff-759b-7a62-09e832f1dcea,BootID:8f294a85-a1ed-4dfb-bdb9-f8818e1f067f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f registry.k8s.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov 23 16:19:23.991: INFO:
Logging kubelet events for node i-0bf2756d84bae9168
Nov 23 16:19:24.061: INFO:
Logging pods the kubelet thinks is on node i-0bf2756d84bae9168
Nov 23 16:19:24.131: INFO: ebs-csi-controller-847c4b8bc4-kwpsb started at 2022-11-23 16:04:56 +0000 UTC (0+5 container statuses recorded)
Nov 23 16:19:24.131: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:19:24.131: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:19:24.131: INFO: Container csi-resizer ready: true, restart count 0
Nov 23 16:19:24.131: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:19:24.131: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:19:24.131: INFO: cilium-scm24 started at 2022-11-23 16:04:35 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:19:24.131: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:19:24.131: INFO: Container cilium-agent ready: true, restart count 0
Nov 23 16:19:24.131: INFO: foo-ctst6 started at 2022-11-23 16:18:59 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:24.131: INFO: Container c ready: true, restart count 0
Nov 23 16:19:24.131: INFO: ebs-csi-node-crshz started at 2022-11-23 16:04:35 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:19:24.131: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:19:24.131: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:19:24.131: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:19:24.397: INFO:
Latency metrics for node i-0bf2756d84bae9168
Nov 23 16:19:24.397: INFO:
Logging node info for node i-0fe7a5031fb82cc5b
Nov 23 16:19:24.462: INFO: Node Info: &Node{ObjectMeta:{i-0fe7a5031fb82cc5b 0297efe3-5070-45f1-8aa2-9a1056b17b59 25782 0 2022-11-23 16:04:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0fe7a5031fb82cc5b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0fe7a5031fb82cc5b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.59.89 csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-5521":"i-0fe7a5031fb82cc5b","ebs.csi.aws.com":"i-0fe7a5031fb82cc5b"} io.cilium.network.ipv4-cilium-host:172.20.57.57 io.cilium.network.ipv4-health-ip:172.20.59.99 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 16:04:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2022-11-23 16:04:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 16:04:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:03 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:19:12 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 16:19:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0fe7a5031fb82cc5b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:03 +0000 UTC,LastTransitionTime:2022-11-23 16:05:03 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:19:22 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:19:22 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:19:22 +0000 UTC,LastTransitionTime:2022-11-23 16:04:33 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:19:22 +0000 UTC,LastTransitionTime:2022-11-23 16:04:54 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.89,},NodeAddress{Type:ExternalIP,Address:34.221.99.167,},NodeAddress{Type:InternalDNS,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0fe7a5031fb82cc5b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-221-99-167.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29292cb2c053c780af3b39ea2fa8c6,SystemUUID:ec29292c-b2c0-53c7-80af-3b39ea2fa8c6,BootID:ff4df73e-ccb2-4109-99ca-04a1b7ab5611,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:b83c1d70989e1fe87583607bf5aee1ee34e52773d4755b95f5cf5a451962f3a4 registry.k8s.io/etcd:3.5.5-0],SizeBytes:102417044,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-5521^90b3fba0-6b4a-11ed-be42-664930db81b2],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-5521^90b3fba0-6b4a-11ed-be42-664930db81b2,DevicePath:,},},Config:nil,},}
Nov 23 16:19:24.463: INFO:
Logging kubelet events for node i-0fe7a5031fb82cc5b
Nov 23 16:19:24.537: INFO:
Logging pods the kubelet thinks is on node i-0fe7a5031fb82cc5b
Nov 23 16:19:24.609: INFO: csi-mockplugin-0 started at 2022-11-23 16:19:06 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:19:24.609: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:19:24.609: INFO: Container driver-registrar ready: true, restart count 0
Nov 23 16:19:24.609: INFO: Container mock ready: true, restart count 0
Nov 23 16:19:24.609: INFO: ebs-csi-controller-847c4b8bc4-lxdkr started at 2022-11-23 16:04:54 +0000 UTC (0+5 container statuses recorded)
Nov 23 16:19:24.609: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:19:24.609: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:19:24.609: INFO: Container csi-resizer ready: true, restart count 0
Nov 23 16:19:24.609: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:19:24.609: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:19:24.609: INFO: pvc-volume-tester-l5df6 started at 2022-11-23 16:19:12 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:24.609: INFO: Container volume-tester ready: true, restart count 0
Nov 23 16:19:24.609: INFO: ebs-csi-node-8ffwl started at 2022-11-23 16:04:34 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:19:24.609: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:19:24.609: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:19:24.609: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:19:24.609: INFO: liveness-d7fa26f5-c898-4d59-aae8-3212a628ddc6 started at 2022-11-23 16:15:49 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:24.609: INFO: Container agnhost-container ready: true, restart count 0
Nov 23 16:19:24.609: INFO: foo-2dgml started at 2022-11-23 16:18:59 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:24.609: INFO: Container c ready: true, restart count 0
Nov 23 16:19:24.609: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 16:19:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:24.609: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:19:24.609: INFO: cilium-727pv started at 2022-11-23 16:04:34 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:19:24.609: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:19:24.609: INFO: Container cilium-agent ready: true, restart count 0
Nov 23 16:19:24.609: INFO: coredns-6c7bddbb75-bpdc6 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:24.609: INFO: Container coredns ready: true, restart count 0
Nov 23 16:19:24.609: INFO: coredns-autoscaler-db7b97744-r4jm7 started at 2022-11-23 16:04:54 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:19:24.609: INFO: Container autoscaler ready: true, restart count 0
Nov 23 16:19:24.911: INFO:
Latency metrics for node i-0fe7a5031fb82cc5b
Nov 23 16:19:24.911: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
�[1mSTEP:�[0m Destroying namespace "fsgroupchangepolicy-4290" for this suite. �[38;5;243m11/23/22 16:19:24.978�[0m
from junit_01.xml
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(OnRootMismatch\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\snew\spod\sfsgroup\sapplied\sto\svolume\scontents$'
test/e2e/storage/testsuites/fsgroupchangepolicy.go:264
k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc0013f3b80, 0xc00317de30, 0x1, {0x0, 0x0}, {0x0, 0x0})
test/e2e/storage/testsuites/fsgroupchangepolicy.go:264 +0x151
k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3()
test/e2e/storage/testsuites/fsgroupchangepolicy.go:235 +0x298
{"msg":"FAILED External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","completed":0,"skipped":3,"failed":1,"failures":["External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents"]}
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
test/e2e/framework/framework.go:186
�[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m11/23/22 16:07:45.665�[0m
Nov 23 16:07:45.665: INFO: >>> kubeConfig: /root/.kube/config
�[1mSTEP:�[0m Building a namespace api object, basename fsgroupchangepolicy �[38;5;243m11/23/22 16:07:45.667�[0m
�[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m11/23/22 16:07:45.858�[0m
�[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m11/23/22 16:07:45.982�[0m
[It] (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
test/e2e/storage/testsuites/fsgroupchangepolicy.go:216
Nov 23 16:07:46.106: INFO: Creating resource for dynamic PV
Nov 23 16:07:46.106: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi}
�[1mSTEP:�[0m creating a StorageClass fsgroupchangepolicy-439-e2e-scgx4fm �[38;5;243m11/23/22 16:07:46.107�[0m
�[1mSTEP:�[0m creating a claim �[38;5;243m11/23/22 16:07:46.17�[0m
Nov 23 16:07:46.170: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
�[1mSTEP:�[0m Creating Pod in namespace fsgroupchangepolicy-439 with fsgroup 1000 �[38;5;243m11/23/22 16:07:46.379�[0m
Nov 23 16:07:46.466: INFO: Waiting up to 5m0s for pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc" in namespace "fsgroupchangepolicy-439" to be "running"
Nov 23 16:07:46.533: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 66.428252ms
Nov 23 16:07:48.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2.130189209s
Nov 23 16:07:50.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4.130527689s
Nov 23 16:07:52.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 6.12990407s
Nov 23 16:07:54.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 8.130020244s
Nov 23 16:07:56.595: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 10.129057865s
Nov 23 16:07:58.598: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 12.131884614s
Nov 23 16:08:00.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 14.130755082s
Nov 23 16:08:02.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 16.129899812s
Nov 23 16:08:04.600: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 18.133718225s
Nov 23 16:08:06.599: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 20.132878939s
Nov 23 16:08:08.598: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 22.13136535s
Nov 23 16:08:10.600: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 24.13381789s
Nov 23 16:08:12.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 26.130941603s
Nov 23 16:08:14.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 28.130859824s
Nov 23 16:08:16.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 30.130678136s
Nov 23 16:08:18.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 32.13094739s
Nov 23 16:08:20.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 34.13087713s
Nov 23 16:08:22.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 36.130295713s
Nov 23 16:08:24.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 38.130652062s
Nov 23 16:08:26.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 40.130059368s
Nov 23 16:08:28.598: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 42.131330387s
Nov 23 16:08:30.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 44.130107549s
Nov 23 16:08:32.605: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 46.138996549s
Nov 23 16:08:34.602: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 48.136039333s
Nov 23 16:08:36.601: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 50.134367293s
Nov 23 16:08:38.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 52.131063317s
Nov 23 16:08:40.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 54.130858547s
Nov 23 16:08:42.598: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 56.131492887s
Nov 23 16:08:44.598: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 58.131757718s
Nov 23 16:08:46.602: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.135747709s
Nov 23 16:08:48.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.13055467s
Nov 23 16:08:50.599: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.132330055s
Nov 23 16:08:52.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.131038373s
Nov 23 16:08:54.610: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.143694957s
Nov 23 16:08:56.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.131118703s
Nov 23 16:08:58.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.130524891s
Nov 23 16:09:00.602: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.135800996s
Nov 23 16:09:02.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.130069335s
Nov 23 16:09:04.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.130075853s
Nov 23 16:09:06.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.130260818s
Nov 23 16:09:08.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.130931498s
Nov 23 16:09:10.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.130306772s
Nov 23 16:09:12.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.129668963s
Nov 23 16:09:14.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.130093719s
Nov 23 16:09:16.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.130203061s
Nov 23 16:09:18.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.130689679s
Nov 23 16:09:20.598: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.132044627s
Nov 23 16:09:22.602: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.136209285s
Nov 23 16:09:24.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m38.130507274s
Nov 23 16:09:26.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m40.130525905s
Nov 23 16:09:28.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m42.130270588s
Nov 23 16:09:30.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.130893724s
Nov 23 16:09:32.598: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.13213364s
Nov 23 16:09:34.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.129959616s
Nov 23 16:09:36.598: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m50.131290591s
Nov 23 16:09:38.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m52.130383664s
Nov 23 16:09:40.601: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m54.135011919s
Nov 23 16:09:42.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m56.130172832s
Nov 23 16:09:44.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 1m58.130023168s
Nov 23 16:09:46.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m0.130390023s
Nov 23 16:09:48.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m2.131160553s
Nov 23 16:09:50.600: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m4.134208145s
Nov 23 16:09:52.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.130254931s
Nov 23 16:09:54.599: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.132615246s
Nov 23 16:09:56.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.129840738s
Nov 23 16:09:58.598: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.131796862s
Nov 23 16:10:00.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.13069962s
Nov 23 16:10:02.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.129667362s
Nov 23 16:10:04.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.130072769s
Nov 23 16:10:06.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.13060402s
Nov 23 16:10:08.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.130843875s
Nov 23 16:10:10.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.131191654s
Nov 23 16:10:12.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.129591916s
Nov 23 16:10:14.629: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.162542723s
Nov 23 16:10:16.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.130551124s
Nov 23 16:10:18.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.130666693s
Nov 23 16:10:20.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.130544663s
Nov 23 16:10:22.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.129772772s
Nov 23 16:10:24.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.130368149s
Nov 23 16:10:26.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.130552808s
Nov 23 16:10:28.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.130401697s
Nov 23 16:10:30.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.130943648s
Nov 23 16:10:32.599: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.132341792s
Nov 23 16:10:34.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.130420201s
Nov 23 16:10:36.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.129932527s
Nov 23 16:10:38.598: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.131625298s
Nov 23 16:10:40.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.130392525s
Nov 23 16:10:42.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.130134059s
Nov 23 16:10:44.603: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 2m58.136583416s
Nov 23 16:10:46.602: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m0.13554385s
Nov 23 16:10:48.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m2.130545376s
Nov 23 16:10:50.608: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.142134918s
Nov 23 16:10:52.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m6.131048347s
Nov 23 16:10:54.599: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m8.132456427s
Nov 23 16:10:56.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.130635398s
Nov 23 16:10:58.599: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.133175147s
Nov 23 16:11:00.598: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m14.131726347s
Nov 23 16:11:02.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m16.130000497s
Nov 23 16:11:04.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m18.130838515s
Nov 23 16:11:06.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m20.130135871s
Nov 23 16:11:08.598: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m22.132221953s
Nov 23 16:11:10.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.13037985s
Nov 23 16:11:12.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.129799391s
Nov 23 16:11:14.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.130447182s
Nov 23 16:11:16.599: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.132371548s
Nov 23 16:11:18.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.130210818s
Nov 23 16:11:20.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m34.130097887s
Nov 23 16:11:22.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m36.130982242s
Nov 23 16:11:24.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m38.129496427s
Nov 23 16:11:26.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m40.130393234s
Nov 23 16:11:28.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m42.131249385s
Nov 23 16:11:30.599: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m44.132598004s
Nov 23 16:11:32.606: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m46.140178005s
Nov 23 16:11:34.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m48.130020543s
Nov 23 16:11:36.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m50.129537609s
Nov 23 16:11:38.598: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m52.131361061s
Nov 23 16:11:40.598: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m54.131304116s
Nov 23 16:11:42.600: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m56.133712228s
Nov 23 16:11:44.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 3m58.131036675s
Nov 23 16:11:46.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m0.130786061s
Nov 23 16:11:48.600: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m2.133965307s
Nov 23 16:11:50.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m4.130609219s
Nov 23 16:11:52.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.129790104s
Nov 23 16:11:54.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.12993206s
Nov 23 16:11:56.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.130466997s
Nov 23 16:11:58.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.129914813s
Nov 23 16:12:00.598: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.131889256s
Nov 23 16:12:02.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m16.130090739s
Nov 23 16:12:04.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m18.130747381s
Nov 23 16:12:06.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m20.130924521s
Nov 23 16:12:08.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m22.130872316s
Nov 23 16:12:10.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m24.130277098s
Nov 23 16:12:12.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.130358292s
Nov 23 16:12:14.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.129893736s
Nov 23 16:12:16.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.129670386s
Nov 23 16:12:18.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.1309408s
Nov 23 16:12:20.602: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.135576553s
Nov 23 16:12:22.599: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.132344341s
Nov 23 16:12:24.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.130453655s
Nov 23 16:12:26.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.130369945s
Nov 23 16:12:28.595: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.1292826s
Nov 23 16:12:30.609: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.142295688s
Nov 23 16:12:32.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.130065196s
Nov 23 16:12:34.657: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.190365825s
Nov 23 16:12:36.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.130955716s
Nov 23 16:12:38.596: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.130161809s
Nov 23 16:12:40.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.13126871s
Nov 23 16:12:42.614: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.147983478s
Nov 23 16:12:44.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.130843351s
Nov 23 16:12:46.597: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.130993528s
Nov 23 16:12:46.661: INFO: Pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.19518742s
Nov 23 16:12:46.661: INFO: Unexpected error:
<*errors.errorString | 0xc000e83530>: {
s: "pod \"pod-7732871b-ce9e-4b17-a498-2a87a01f48fc\" is not Running: timed out while waiting for pod fsgroupchangepolicy-439/pod-7732871b-ce9e-4b17-a498-2a87a01f48fc to be running",
}
Nov 23 16:12:46.662: FAIL: pod "pod-7732871b-ce9e-4b17-a498-2a87a01f48fc" is not Running: timed out while waiting for pod fsgroupchangepolicy-439/pod-7732871b-ce9e-4b17-a498-2a87a01f48fc to be running
Full Stack Trace
k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc0013f3b80, 0xc00317de30, 0x1, {0x0, 0x0}, {0x0, 0x0})
test/e2e/storage/testsuites/fsgroupchangepolicy.go:264 +0x151
k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3()
test/e2e/storage/testsuites/fsgroupchangepolicy.go:235 +0x298
�[1mSTEP:�[0m Deleting pvc �[38;5;243m11/23/22 16:12:46.662�[0m
Nov 23 16:12:46.725: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comjnvqx"
�[1mSTEP:�[0m Deleting sc �[38;5;243m11/23/22 16:12:46.799�[0m
[AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
test/e2e/framework/framework.go:187
�[1mSTEP:�[0m Collecting events from namespace "fsgroupchangepolicy-439". �[38;5;243m11/23/22 16:12:46.863�[0m
�[1mSTEP:�[0m Found 5 events. �[38;5;243m11/23/22 16:12:46.926�[0m
Nov 23 16:12:46.926: INFO: At 2022-11-23 16:07:46 +0000 UTC - event for ebs.csi.aws.comjnvqx: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
Nov 23 16:12:46.926: INFO: At 2022-11-23 16:07:46 +0000 UTC - event for ebs.csi.aws.comjnvqx: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-439/ebs.csi.aws.comjnvqx"
Nov 23 16:12:46.926: INFO: At 2022-11-23 16:07:46 +0000 UTC - event for ebs.csi.aws.comjnvqx: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
Nov 23 16:12:46.926: INFO: At 2022-11-23 16:07:56 +0000 UTC - event for ebs.csi.aws.comjnvqx: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-439-e2e-scgx4fm": rpc error: code = DeadlineExceeded desc = context deadline exceeded
Nov 23 16:12:46.926: INFO: At 2022-11-23 16:08:51 +0000 UTC - event for ebs.csi.aws.comjnvqx: {ebs.csi.aws.com_ebs-csi-controller-847c4b8bc4-kwpsb_1a2cbd8b-0cf5-4c90-90ce-f09bda3f8c41 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-439-e2e-scgx4fm": rpc error: code = Internal desc = Could not create volume "pvc-b3ff603c-4b9f-4d7e-89a4-e60e132bf99b": could not create volume in EC2: RequestCanceled: request context canceled
caused by: context deadline exceeded
Nov 23 16:12:46.989: INFO: POD NODE PHASE GRACE CONDITIONS
Nov 23 16:12:46.989: INFO: pod-7732871b-ce9e-4b17-a498-2a87a01f48fc Pending []
Nov 23 16:12:46.989: INFO:
Nov 23 16:12:47.242: INFO:
Logging node info for node i-060d86c6ffc5622b0
Nov 23 16:12:47.306: INFO: Node Info: &Node{ObjectMeta:{i-060d86c6ffc5622b0 b49ff0b3-8686-4c0f-b991-1d427ca43e2a 7999 0 2022-11-23 16:02:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-060d86c6ffc5622b0 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.40.57 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-060d86c6ffc5622b0"} io.cilium.network.ipv4-cilium-host:172.20.57.204 io.cilium.network.ipv4-health-ip:172.20.37.200 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 16:02:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2022-11-23 16:03:01 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:03:21 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 16:03:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 16:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:10:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-060d86c6ffc5622b0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:04:42 +0000 UTC,LastTransitionTime:2022-11-23 16:04:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:10:12 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:10:12 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:10:12 +0000 UTC,LastTransitionTime:2022-11-23 16:02:30 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:10:12 +0000 UTC,LastTransitionTime:2022-11-23 16:03:24 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.40.57,},NodeAddress{Type:ExternalIP,Address:54.200.161.23,},NodeAddress{Type:InternalDNS,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-060d86c6ffc5622b0.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-200-161-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec226e0d77c347467223a292b33eb0c5,SystemUUID:ec226e0d-77c3-4746-7223-a292b33eb0c5,BootID:936b9dd2-17e7-49ac-8c1e-7556bf2c9ee8,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.25.4],SizeBytes:129085904,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.25.4],SizeBytes:118436153,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.25.4],SizeBytes:51921208,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42928334,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41477333,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:85d3f1e9dacc72531445989bb10999e1e70ebc409d11be57e5baa5f031a893b0 registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.1],SizeBytes:18257577,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955428,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov 23 16:12:47.307: INFO:
Logging kubelet events for node i-060d86c6ffc5622b0
Nov 23 16:12:47.381: INFO:
Logging pods the kubelet thinks is on node i-060d86c6ffc5622b0
Nov 23 16:12:47.518: INFO: etcd-manager-events-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:12:47.518: INFO: Container etcd-manager ready: true, restart count 0
Nov 23 16:12:47.518: INFO: kops-controller-7w98n started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:12:47.518: INFO: Container kops-controller ready: true, restart count 0
Nov 23 16:12:47.518: INFO: cilium-operator-7f7bcdf787-xnz2q started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:12:47.518: INFO: Container cilium-operator ready: true, restart count 1
Nov 23 16:12:47.518: INFO: aws-cloud-controller-manager-dzzkp started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:12:47.518: INFO: Container aws-cloud-controller-manager ready: true, restart count 0
Nov 23 16:12:47.518: INFO: etcd-manager-main-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:12:47.518: INFO: Container etcd-manager ready: true, restart count 0
Nov 23 16:12:47.518: INFO: kube-apiserver-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+2 container statuses recorded)
Nov 23 16:12:47.518: INFO: Container healthcheck ready: true, restart count 0
Nov 23 16:12:47.518: INFO: Container kube-apiserver ready: true, restart count 1
Nov 23 16:12:47.518: INFO: kube-controller-manager-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:12:47.518: INFO: Container kube-controller-manager ready: true, restart count 2
Nov 23 16:12:47.518: INFO: kube-scheduler-i-060d86c6ffc5622b0 started at 2022-11-23 16:02:01 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:12:47.518: INFO: Container kube-scheduler ready: true, restart count 0
Nov 23 16:12:47.518: INFO: dns-controller-89c6d5f67-9hm78 started at 2022-11-23 16:03:06 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:12:47.518: INFO: Container dns-controller ready: true, restart count 0
Nov 23 16:12:47.518: INFO: cilium-f2gg7 started at 2022-11-23 16:03:06 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:12:47.518: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:12:47.518: INFO: Container cilium-agent ready: true, restart count 1
Nov 23 16:12:47.518: INFO: ebs-csi-node-vpv4t started at 2022-11-23 16:03:06 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:12:47.518: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:12:47.518: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:12:47.518: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:12:47.799: INFO:
Latency metrics for node i-060d86c6ffc5622b0
Nov 23 16:12:47.799: INFO:
Logging node info for node i-0765e732a9635923b
Nov 23 16:12:47.862: INFO: Node Info: &Node{ObjectMeta:{i-0765e732a9635923b 10981661-cc27-412a-9e66-7a37930b898f 14279 0 2022-11-23 16:04:39 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0765e732a9635923b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0765e732a9635923b topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.58.74 csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-5280":"i-0765e732a9635923b","csi-mock-csi-mock-volumes-2288":"csi-mock-csi-mock-volumes-2288","ebs.csi.aws.com":"i-0765e732a9635923b"} io.cilium.network.ipv4-cilium-host:172.20.54.198 io.cilium.network.ipv4-health-ip:172.20.60.24 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:07 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:12:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0765e732a9635923b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 16:05:07 +0000 UTC,LastTransitionTime:2022-11-23 16:05:07 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 16:11:38 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 16:11:38 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 16:11:38 +0000 UTC,LastTransitionTime:2022-11-23 16:04:39 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 16:11:38 +0000 UTC,LastTransitionTime:2022-11-23 16:05:00 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.58.74,},NodeAddress{Type:ExternalIP,Address:52.25.125.23,},NodeAddress{Type:InternalDNS,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0765e732a9635923b.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-25-125-23.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec23d129b4036d8d979053444dfae008,SystemUUID:ec23d129-b403-6d8d-9790-53444dfae008,BootID:fe390540-4601-456e-9686-93a36ea3dc8f,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.10,KubeletVersion:v1.25.4,KubeProxyVersion:v1.25.4,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.4],SizeBytes:63279305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Nov 23 16:12:47.862: INFO:
Logging kubelet events for node i-0765e732a9635923b
Nov 23 16:12:47.928: INFO:
Logging pods the kubelet thinks is on node i-0765e732a9635923b
Nov 23 16:12:48.065: INFO: csi-mockplugin-0 started at 2022-11-23 16:12:39 +0000 UTC (0+4 container statuses recorded)
Nov 23 16:12:48.065: INFO: Container busybox ready: true, restart count 0
Nov 23 16:12:48.065: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:12:48.065: INFO: Container driver-registrar ready: true, restart count 0
Nov 23 16:12:48.065: INFO: Container mock ready: true, restart count 0
Nov 23 16:12:48.065: INFO: cilium-czwdq started at 2022-11-23 16:04:40 +0000 UTC (1+1 container statuses recorded)
Nov 23 16:12:48.065: INFO: Init container clean-cilium-state ready: true, restart count 0
Nov 23 16:12:48.065: INFO: Container cilium-agent ready: true, restart count 0
Nov 23 16:12:48.065: INFO: coredns-6c7bddbb75-2n2dw started at 2022-11-23 16:05:19 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:12:48.065: INFO: Container coredns ready: true, restart count 0
Nov 23 16:12:48.065: INFO: inline-volume-tester-4kf29 started at 2022-11-23 16:12:20 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:12:48.065: INFO: Container csi-volume-tester ready: true, restart count 0
Nov 23 16:12:48.065: INFO: ebs-csi-node-5khlc started at 2022-11-23 16:04:40 +0000 UTC (0+3 container statuses recorded)
Nov 23 16:12:48.065: INFO: Container ebs-plugin ready: true, restart count 0
Nov 23 16:12:48.065: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:12:48.065: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:12:48.065: INFO: deployment-shared-map-item-removal-97c6466db-g46pt started at 2022-11-23 16:10:55 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:12:48.065: INFO: Container nginx ready: false, restart count 0
Nov 23 16:12:48.065: INFO: netserver-0 started at 2022-11-23 16:12:18 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:12:48.065: INFO: Container webserver ready: true, restart count 0
Nov 23 16:12:48.065: INFO: netserver-0 started at 2022-11-23 16:12:47 +0000 UTC (0+1 container statuses recorded)
Nov 23 16:12:48.065: INFO: Container webserver ready: false, restart count 0
Nov 23 16:12:48.065: INFO: csi-hostpathplugin-0 started at 2022-11-23 16:12:20 +0000 UTC (0+7 container statuses recorded)
Nov 23 16:12:48.065: INFO: Container csi-attacher ready: true, restart count 0
Nov 23 16:12:48.065: INFO: Container csi-provisioner ready: true, restart count 0
Nov 23 16:12:48.065: INFO: Container csi-resizer ready: true, restart count 0
Nov 23 16:12:48.065: INFO: Container csi-snapshotter ready: true, restart count 0
Nov 23 16:12:48.065: INFO: Container hostpath ready: true, restart count 0
Nov 23 16:12:48.065: INFO: Container liveness-probe ready: true, restart count 0
Nov 23 16:12:48.065: INFO: Container node-driver-registrar ready: true, restart count 0
Nov 23 16:12:48.402: INFO:
Latency metrics for node i-0765e732a9635923b
Nov 23 16:12:48.402: INFO:
Logging node info for node i-0ade3c3082581cb52
Nov 23 16:12:48.467: INFO: Node Info: &Node{ObjectMeta:{i-0ade3c3082581cb52 2305662a-7d05-4979-bcec-220b8b4672db 13497 0 2022-11-23 16:04:42 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0ade3c3082581cb52 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0ade3c3082581cb52 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[alpha.kubernetes.io/provided-node-ip:172.20.56.12 csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0ade3c3082581cb52"} io.cilium.network.ipv4-cilium-host:172.20.45.237 io.cilium.network.ipv4-health-ip:172.20.55.90 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 16:04:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:alpha.kubernetes.io/provided-node-ip":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2022-11-23 16:04:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {cilium-agent Update v1 2022-11-23 16:05:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 16:05:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 16:12:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status"