e2e-kops-grid-cilium-etcd-u2204-k25-ko26 #1619315690945122304 Results

Test Failures

Kubernetes e2e suite [It] [sig-network] Networking Granular Checks: Services should function for multiple endpoint-Services with same selector 5m23s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-network\]\sNetworking\sGranular\sChecks\:\sServices\sshould\sfunction\sfor\smultiple\sendpoint\-Services\swith\ssame\sselector$'

test/e2e/framework/network/utils.go:864
k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).createNetProxyPods(0xc000e7ed20, {0x736e39d, 0x9}, 0xc00270ec00)
	test/e2e/framework/network/utils.go:864 +0x1d0
k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).setupCore(0xc000e7ed20, 0x7f3b5461f1e0?)
	test/e2e/framework/network/utils.go:761 +0x55
k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).setup(0xc000e7ed20, 0x3e?)
	test/e2e/framework/network/utils.go:776 +0x3e
k8s.io/kubernetes/test/e2e/framework/network.NewNetworkingTestConfig(0xc000a769a0, {0x0, 0x0, 0x70000c001f6bd90?})
	test/e2e/framework/network/utils.go:129 +0x125
k8s.io/kubernetes/test/e2e/network.glob..func21.6.10()
	test/e2e/network/networking.go:284 +0x3e
				{"msg":"FAILED [sig-network] Networking Granular Checks: Services should function for multiple endpoint-Services with same selector","completed":21,"skipped":182,"failed":1,"failures":["[sig-network] Networking Granular Checks: Services should function for multiple endpoint-Services with same selector"]}

[BeforeEach] [sig-network] Networking
  test/e2e/framework/framework.go:186
�[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m01/28/23 13:12:46.031�[0m
Jan 28 13:12:46.031: INFO: >>> kubeConfig: /root/.kube/config
�[1mSTEP:�[0m Building a namespace api object, basename nettest �[38;5;243m01/28/23 13:12:46.032�[0m
�[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m01/28/23 13:12:46.683�[0m
�[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m01/28/23 13:12:47.111�[0m
[It] should function for multiple endpoint-Services with same selector
  test/e2e/network/networking.go:283
�[1mSTEP:�[0m Performing setup for networking test in namespace nettest-4274 �[38;5;243m01/28/23 13:12:47.54�[0m
�[1mSTEP:�[0m creating a selector �[38;5;243m01/28/23 13:12:47.541�[0m
�[1mSTEP:�[0m Creating the service pods in kubernetes �[38;5;243m01/28/23 13:12:47.541�[0m
Jan 28 13:12:47.541: INFO: Waiting up to 10m0s for all (but 0) nodes to be schedulable
Jan 28 13:12:48.843: INFO: Waiting up to 5m0s for pod "netserver-0" in namespace "nettest-4274" to be "running and ready"
Jan 28 13:12:49.058: INFO: Pod "netserver-0": Phase="Pending", Reason="", readiness=false. Elapsed: 214.411716ms
Jan 28 13:12:49.058: INFO: The phase of Pod netserver-0 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:12:51.280: INFO: Pod "netserver-0": Phase="Running", Reason="", readiness=false. Elapsed: 2.436496959s
Jan 28 13:12:51.280: INFO: The phase of Pod netserver-0 is Running (Ready = false)
Jan 28 13:12:53.274: INFO: Pod "netserver-0": Phase="Running", Reason="", readiness=false. Elapsed: 4.430592878s
Jan 28 13:12:53.274: INFO: The phase of Pod netserver-0 is Running (Ready = false)
Jan 28 13:12:55.274: INFO: Pod "netserver-0": Phase="Running", Reason="", readiness=false. Elapsed: 6.430574535s
Jan 28 13:12:55.274: INFO: The phase of Pod netserver-0 is Running (Ready = false)
Jan 28 13:12:57.274: INFO: Pod "netserver-0": Phase="Running", Reason="", readiness=false. Elapsed: 8.430578205s
Jan 28 13:12:57.274: INFO: The phase of Pod netserver-0 is Running (Ready = false)
Jan 28 13:12:59.273: INFO: Pod "netserver-0": Phase="Running", Reason="", readiness=true. Elapsed: 10.430000509s
Jan 28 13:12:59.273: INFO: The phase of Pod netserver-0 is Running (Ready = true)
Jan 28 13:12:59.273: INFO: Pod "netserver-0" satisfied condition "running and ready"
Jan 28 13:12:59.488: INFO: Waiting up to 5m0s for pod "netserver-1" in namespace "nettest-4274" to be "running and ready"
Jan 28 13:12:59.704: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 216.021626ms
Jan 28 13:12:59.704: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:01.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2.430799397s
Jan 28 13:13:01.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:03.924: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4.435830847s
Jan 28 13:13:03.924: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:05.924: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 6.436633422s
Jan 28 13:13:05.925: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:07.921: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 8.433019101s
Jan 28 13:13:07.921: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:09.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 10.431654259s
Jan 28 13:13:09.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:11.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 12.431510484s
Jan 28 13:13:11.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:13.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 14.431077048s
Jan 28 13:13:13.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:15.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 16.431445474s
Jan 28 13:13:15.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:17.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 18.431666031s
Jan 28 13:13:17.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:19.922: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 20.434542253s
Jan 28 13:13:19.922: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:21.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 22.432492322s
Jan 28 13:13:21.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:23.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 24.43096747s
Jan 28 13:13:23.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:25.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 26.431483002s
Jan 28 13:13:25.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:27.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 28.431025224s
Jan 28 13:13:27.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:29.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 30.43216353s
Jan 28 13:13:29.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:31.922: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 32.433655181s
Jan 28 13:13:31.922: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:33.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 34.431613009s
Jan 28 13:13:33.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:35.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 36.432467006s
Jan 28 13:13:35.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:37.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 38.432247536s
Jan 28 13:13:37.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:39.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 40.432029564s
Jan 28 13:13:39.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:41.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 42.431251644s
Jan 28 13:13:41.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:43.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 44.431786022s
Jan 28 13:13:43.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:45.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 46.432515948s
Jan 28 13:13:45.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:47.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 48.43179739s
Jan 28 13:13:47.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:49.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 50.431400535s
Jan 28 13:13:49.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:51.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 52.43207564s
Jan 28 13:13:51.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:53.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 54.431567617s
Jan 28 13:13:53.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:55.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 56.432007916s
Jan 28 13:13:55.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:57.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 58.431963834s
Jan 28 13:13:57.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:13:59.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.431822867s
Jan 28 13:13:59.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:01.921: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.433535033s
Jan 28 13:14:01.921: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:03.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.431322344s
Jan 28 13:14:03.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:05.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.431989294s
Jan 28 13:14:05.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:07.927: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.43949696s
Jan 28 13:14:07.927: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:09.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.43199818s
Jan 28 13:14:09.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:11.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.432169095s
Jan 28 13:14:11.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:13.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.431975194s
Jan 28 13:14:13.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:15.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.432055139s
Jan 28 13:14:15.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:17.923: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.43522273s
Jan 28 13:14:17.923: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:19.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.431148152s
Jan 28 13:14:19.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:21.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.431480713s
Jan 28 13:14:21.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:23.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.431003536s
Jan 28 13:14:23.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:25.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.43112794s
Jan 28 13:14:25.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:27.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.43165281s
Jan 28 13:14:27.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:29.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.431621252s
Jan 28 13:14:29.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:31.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.431262948s
Jan 28 13:14:31.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:33.924: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.436358565s
Jan 28 13:14:33.924: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:35.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.43188133s
Jan 28 13:14:35.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:37.921: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m38.43285753s
Jan 28 13:14:37.921: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:39.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m40.432231505s
Jan 28 13:14:39.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:41.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m42.431552763s
Jan 28 13:14:41.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:43.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.431870107s
Jan 28 13:14:43.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:45.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.431141323s
Jan 28 13:14:45.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:47.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.432117129s
Jan 28 13:14:47.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:49.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m50.431890871s
Jan 28 13:14:49.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:51.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m52.432050939s
Jan 28 13:14:51.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:53.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m54.43211916s
Jan 28 13:14:53.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:55.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m56.431084452s
Jan 28 13:14:55.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:57.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 1m58.43252865s
Jan 28 13:14:57.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:14:59.921: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m0.433343186s
Jan 28 13:14:59.921: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:01.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m2.431001959s
Jan 28 13:15:01.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:03.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m4.430990675s
Jan 28 13:15:03.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:05.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.431402844s
Jan 28 13:15:05.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:07.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.432243263s
Jan 28 13:15:07.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:09.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.431298466s
Jan 28 13:15:09.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:11.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.431269215s
Jan 28 13:15:11.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:13.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.43219388s
Jan 28 13:15:13.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:15.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.431682556s
Jan 28 13:15:15.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:17.921: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.433072902s
Jan 28 13:15:17.921: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:19.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.431759116s
Jan 28 13:15:19.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:21.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.43201036s
Jan 28 13:15:21.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:23.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.431622001s
Jan 28 13:15:23.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:25.922: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.433752981s
Jan 28 13:15:25.922: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:27.921: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.432966904s
Jan 28 13:15:27.921: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:29.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.431661302s
Jan 28 13:15:29.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:31.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.431847387s
Jan 28 13:15:31.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:33.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.431886414s
Jan 28 13:15:33.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:35.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.431994784s
Jan 28 13:15:35.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:37.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.431284402s
Jan 28 13:15:37.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:39.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.431648463s
Jan 28 13:15:39.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:41.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.432053497s
Jan 28 13:15:41.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:43.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.432269154s
Jan 28 13:15:43.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:45.921: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.432705762s
Jan 28 13:15:45.921: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:47.925: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.436653727s
Jan 28 13:15:47.925: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:49.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.431417945s
Jan 28 13:15:49.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:51.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.431022087s
Jan 28 13:15:51.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:53.921: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.433281331s
Jan 28 13:15:53.921: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:55.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.43126046s
Jan 28 13:15:55.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:57.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 2m58.431867989s
Jan 28 13:15:57.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:15:59.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m0.431639661s
Jan 28 13:15:59.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:01.923: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m2.434974056s
Jan 28 13:16:01.923: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:03.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.432406593s
Jan 28 13:16:03.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:05.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m6.431958358s
Jan 28 13:16:05.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:07.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m8.431168564s
Jan 28 13:16:07.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:09.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.430833209s
Jan 28 13:16:09.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:11.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.431225808s
Jan 28 13:16:11.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:13.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m14.431960348s
Jan 28 13:16:13.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:15.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m16.431901782s
Jan 28 13:16:15.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:17.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m18.432456709s
Jan 28 13:16:17.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:19.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m20.431841929s
Jan 28 13:16:19.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:21.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m22.43138525s
Jan 28 13:16:21.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:23.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.432036741s
Jan 28 13:16:23.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:25.921: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.433141183s
Jan 28 13:16:25.921: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:27.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.431681346s
Jan 28 13:16:27.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:29.927: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.439408877s
Jan 28 13:16:29.927: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:31.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.431714591s
Jan 28 13:16:31.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:33.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m34.432385917s
Jan 28 13:16:33.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:35.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m36.430965297s
Jan 28 13:16:35.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:37.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m38.431715867s
Jan 28 13:16:37.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:39.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m40.431859873s
Jan 28 13:16:39.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:41.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m42.431155745s
Jan 28 13:16:41.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:43.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m44.432343875s
Jan 28 13:16:43.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:45.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m46.431903687s
Jan 28 13:16:45.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:47.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m48.430970459s
Jan 28 13:16:47.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:49.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m50.432229267s
Jan 28 13:16:49.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:51.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m52.431075396s
Jan 28 13:16:51.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:53.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m54.43231755s
Jan 28 13:16:53.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:55.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m56.430740469s
Jan 28 13:16:55.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:57.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 3m58.431039942s
Jan 28 13:16:57.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:16:59.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m0.432047748s
Jan 28 13:16:59.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:01.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m2.431528134s
Jan 28 13:17:01.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:03.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m4.431561274s
Jan 28 13:17:03.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:05.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.43190532s
Jan 28 13:17:05.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:07.921: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.432757882s
Jan 28 13:17:07.921: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:09.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.431689921s
Jan 28 13:17:09.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:11.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.43204582s
Jan 28 13:17:11.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:13.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.431753041s
Jan 28 13:17:13.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:15.921: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m16.432739795s
Jan 28 13:17:15.921: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:17.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m18.430780818s
Jan 28 13:17:17.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:19.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m20.431810519s
Jan 28 13:17:19.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:21.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m22.432117314s
Jan 28 13:17:21.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:23.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m24.431073758s
Jan 28 13:17:23.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:25.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.432369936s
Jan 28 13:17:25.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:27.973: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.484748659s
Jan 28 13:17:27.973: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:29.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.431644265s
Jan 28 13:17:29.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:31.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.431759282s
Jan 28 13:17:31.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:33.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.432147954s
Jan 28 13:17:33.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:35.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.431767857s
Jan 28 13:17:35.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:37.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.432269652s
Jan 28 13:17:37.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:39.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.430965677s
Jan 28 13:17:39.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:41.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.431291751s
Jan 28 13:17:41.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:43.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.432098229s
Jan 28 13:17:43.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:45.919: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.431134306s
Jan 28 13:17:45.919: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:47.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.431662726s
Jan 28 13:17:47.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:49.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.432039604s
Jan 28 13:17:49.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:51.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.431916278s
Jan 28 13:17:51.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:53.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.432126897s
Jan 28 13:17:53.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:55.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.432153392s
Jan 28 13:17:55.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:57.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.431855132s
Jan 28 13:17:57.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:17:59.920: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.43183639s
Jan 28 13:17:59.920: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:18:00.135: INFO: Pod "netserver-1": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.646732697s
Jan 28 13:18:00.135: INFO: The phase of Pod netserver-1 is Pending, waiting for it to be Running (with Ready = true)
Jan 28 13:18:00.136: INFO: Unexpected error: 
    <*pod.timeoutError | 0xc0037af5f0>: {
        msg: "timed out while waiting for pod nettest-4274/netserver-1 to be running and ready",
        observedObjects: [
            <*v1.Pod | 0xc001948800>{
                TypeMeta: {Kind: "", APIVersion: ""},
                ObjectMeta: {
                    Name: "netserver-1",
                    GenerateName: "",
                    Namespace: "nettest-4274",
                    SelfLink: "",
                    UID: "d181ea23-27f5-4361-9769-f40f162fbee4",
                    ResourceVersion: "38112",
                    Generation: 0,
                    CreationTimestamp: {
                        Time: {
                            wall: 0,
                            ext: 63810508368,
                            loc: {
                                name: "Local",
                                zone: [
                                    {name: "UTC", offset: 0, isDST: false},
                                ],
                                tx: [
                                    {
                                        when: -9223372036854775808,
                                        index: 0,
                                        isstd: false,
                                        isutc: false,
                                    },
                                ],
                                extend: "UTC0",
                                cacheStart: 9223372036854775807,
                                cacheEnd: 9223372036854775807,
                                cacheZone: {name: "UTC", offset: 0, isDST: false},
                            },
                        },
                    },
                    DeletionTimestamp: nil,
                    DeletionGracePeriodSeconds: nil,
                    Labels: {
                        "selector-02c66ecb-dccf-4b64-a54f-bdcd1a423f2f": "true",
                    },
                    Annotations: nil,
                    OwnerReferences: nil,
                    Finalizers: nil,
                    ManagedFields: [
                        {
                            Manager: "e2e.test",
                            Operation: "Update",
                            APIVersion: "v1",
                            Time: {
                                Time: {
                                    wall: 0,
                                    ext: 63810508368,
                                    loc: {
                                        name: "Local",
                                        zone: [...],
                                        tx: [...],
                                        extend: "UTC0",
                                        cacheStart: 9223372036854775807,
                                        cacheEnd: 9223372036854775807,
                                        cacheZone: {name: ..., offset: ..., isDST: ...},
                                    },
                                },
                            },
                            FieldsType: "FieldsV1",
                            FieldsV1: {
                                Raw: "{\"f:metadata\":{\"f:labels\":{\".\":{},\"f:selector-02c66ecb-dccf-4b64-a54f-bdcd1a423f2f\":{}}},\"f:spec\":{\"f:containers\":{\"k:{\\\"name\\\":\\\"webserver\\\"}\":{\".\":{},\"f:args\":{},\"f:image\":{},\"f:imagePullPolicy\":{},\"f:livenessProbe\":{\".\":{},\"f:failureThreshold\":{},\"f:httpGet\":{\".\":{},\"f:path\":{},\"f:port\":{},\"f:scheme\":{}},\"f:initialDelaySeconds\":{},\"f:periodSeconds\":{},\"f:successThreshold\":{},\"f:timeoutSeconds\":{}},\"f:name\":{},\"f:ports\":{\".\":{},\"k:{\\\"containerPort\\\":8081,\\\"protocol\\\":\\\"UDP\\\"}\":{\".\":{},\"f:containerPort\":{},\"f:name\":{},\"f:protocol\":{}},\"k:{\\\"containerPort\\\":8083,\\\"protocol\\\":\\\"TCP\\\"}\":{\".\":{},\"f:containerPort\":{},\"f:name\":{},\"f:protocol\":{}}},\"f:readinessProbe\":{\".\":{},\"f:failureThreshold\":{},\"f:httpGet\":{\".\":{},\"f:path\":{},\"f:port\":{},\"f:scheme\":{}},\"f:...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

Jan 28 13:18:00.136: FAIL: timed out while waiting for pod nettest-4274/netserver-1 to be running and ready

Full Stack Trace
k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).createNetProxyPods(0xc000e7ed20, {0x736e39d, 0x9}, 0xc00270ec00)
	test/e2e/framework/network/utils.go:864 +0x1d0
k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).setupCore(0xc000e7ed20, 0x7f3b5461f1e0?)
	test/e2e/framework/network/utils.go:761 +0x55
k8s.io/kubernetes/test/e2e/framework/network.(*NetworkingTestConfig).setup(0xc000e7ed20, 0x3e?)
	test/e2e/framework/network/utils.go:776 +0x3e
k8s.io/kubernetes/test/e2e/framework/network.NewNetworkingTestConfig(0xc000a769a0, {0x0, 0x0, 0x70000c001f6bd90?})
	test/e2e/framework/network/utils.go:129 +0x125
k8s.io/kubernetes/test/e2e/network.glob..func21.6.10()
	test/e2e/network/networking.go:284 +0x3e
[AfterEach] [sig-network] Networking
  test/e2e/framework/framework.go:187
�[1mSTEP:�[0m Collecting events from namespace "nettest-4274". �[38;5;243m01/28/23 13:18:00.136�[0m
�[1mSTEP:�[0m Found 15 events. �[38;5;243m01/28/23 13:18:00.351�[0m
Jan 28 13:18:00.351: INFO: At 2023-01-28 13:12:48 +0000 UTC - event for netserver-0: {kubelet i-00e83413dfa35ee6b} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.40" already present on machine
Jan 28 13:18:00.351: INFO: At 2023-01-28 13:12:48 +0000 UTC - event for netserver-0: {kubelet i-00e83413dfa35ee6b} Created: Created container webserver
Jan 28 13:18:00.351: INFO: At 2023-01-28 13:12:48 +0000 UTC - event for netserver-0: {kubelet i-00e83413dfa35ee6b} Started: Started container webserver
Jan 28 13:18:00.351: INFO: At 2023-01-28 13:12:48 +0000 UTC - event for netserver-0: {default-scheduler } Scheduled: Successfully assigned nettest-4274/netserver-0 to i-00e83413dfa35ee6b
Jan 28 13:18:00.351: INFO: At 2023-01-28 13:12:48 +0000 UTC - event for netserver-1: {default-scheduler } Scheduled: Successfully assigned nettest-4274/netserver-1 to i-011ed997f238bea92
Jan 28 13:18:00.351: INFO: At 2023-01-28 13:12:48 +0000 UTC - event for netserver-2: {default-scheduler } Scheduled: Successfully assigned nettest-4274/netserver-2 to i-04736cc52ac74b9d6
Jan 28 13:18:00.351: INFO: At 2023-01-28 13:12:48 +0000 UTC - event for netserver-3: {default-scheduler } Scheduled: Successfully assigned nettest-4274/netserver-3 to i-0cceed973097eb227
Jan 28 13:18:00.351: INFO: At 2023-01-28 13:12:49 +0000 UTC - event for netserver-2: {kubelet i-04736cc52ac74b9d6} Created: Created container webserver
Jan 28 13:18:00.352: INFO: At 2023-01-28 13:12:49 +0000 UTC - event for netserver-2: {kubelet i-04736cc52ac74b9d6} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.40" already present on machine
Jan 28 13:18:00.352: INFO: At 2023-01-28 13:12:49 +0000 UTC - event for netserver-2: {kubelet i-04736cc52ac74b9d6} Started: Started container webserver
Jan 28 13:18:00.352: INFO: At 2023-01-28 13:12:49 +0000 UTC - event for netserver-3: {kubelet i-0cceed973097eb227} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.40" already present on machine
Jan 28 13:18:00.352: INFO: At 2023-01-28 13:12:49 +0000 UTC - event for netserver-3: {kubelet i-0cceed973097eb227} Created: Created container webserver
Jan 28 13:18:00.352: INFO: At 2023-01-28 13:12:49 +0000 UTC - event for netserver-3: {kubelet i-0cceed973097eb227} Started: Started container webserver
Jan 28 13:18:00.352: INFO: At 2023-01-28 13:13:48 +0000 UTC - event for netserver-1: {kubelet i-011ed997f238bea92} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d078e19d88f08dab5f54e5b5303e564127c910184e060a3593006e0072f6f053": plugin type="cilium-cni" name="cilium" failed (add): unable to connect to Cilium daemon: failed to create cilium agent client after 30.000000 seconds timeout: Get "http:///var/run/cilium/cilium.sock/v1/config": dial unix /var/run/cilium/cilium.sock: connect: connection refused
Is the agent running?
Jan 28 13:18:00.352: INFO: At 2023-01-28 13:13:49 +0000 UTC - event for netserver-1: {kubelet i-011ed997f238bea92} SandboxChanged: Pod sandbox changed, it will be killed and re-created.
Jan 28 13:18:00.567: INFO: POD          NODE                 PHASE    GRACE  CONDITIONS
Jan 28 13:18:00.567: INFO: netserver-0  i-00e83413dfa35ee6b  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:48 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:58 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:58 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:48 +0000 UTC  }]
Jan 28 13:18:00.567: INFO: netserver-1  i-011ed997f238bea92  Pending         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:48 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:48 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:48 +0000 UTC ContainersNotReady containers with unready status: [webserver]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:48 +0000 UTC  }]
Jan 28 13:18:00.567: INFO: netserver-2  i-04736cc52ac74b9d6  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:48 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:59 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:59 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:48 +0000 UTC  }]
Jan 28 13:18:00.567: INFO: netserver-3  i-0cceed973097eb227  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:48 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:59 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:59 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:12:48 +0000 UTC  }]
Jan 28 13:18:00.567: INFO: 
Jan 28 13:18:01.026: INFO: Unable to fetch nettest-4274/netserver-1/webserver logs: the server rejected our request for an unknown reason (get pods netserver-1)
Jan 28 13:18:01.680: INFO: 
Logging node info for node i-00e83413dfa35ee6b
Jan 28 13:18:01.895: INFO: Node Info: &Node{ObjectMeta:{i-00e83413dfa35ee6b    1a086b23-20ac-414b-98eb-191b96574ca7 45888 0 2023-01-28 12:52:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-00e83413dfa35ee6b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-00e83413dfa35ee6b topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9664":"i-00e83413dfa35ee6b","ebs.csi.aws.com":"i-00e83413dfa35ee6b"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 12:52:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 12:52:27 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 12:52:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 12:52:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 12:52:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-01-28 13:17:12 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 13:18:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-00e83413dfa35ee6b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:18 +0000 UTC,LastTransitionTime:2023-01-28 12:52:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:18 +0000 UTC,LastTransitionTime:2023-01-28 12:52:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:18 +0000 UTC,LastTransitionTime:2023-01-28 12:52:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 13:17:18 +0000 UTC,LastTransitionTime:2023-01-28 12:52:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.66,},NodeAddress{Type:ExternalIP,Address:54.169.3.124,},NodeAddress{Type:InternalDNS,Address:i-00e83413dfa35ee6b.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-00e83413dfa35ee6b.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-169-3-124.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dd38b50ae71201c15829db9e8dbeb,SystemUUID:ec2dd38b-50ae-7120-1c15-829db9e8dbeb,BootID:603bf541-7f58-4980-9ee5-4bcd00e1d1ef,KernelVersion:5.15.0-1028-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.25.6,KubeProxyVersion:v1.25.6,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.6],SizeBytes:63291593,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c registry.k8s.io/e2e-test-images/nautilus:1.5],SizeBytes:49642095,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 registry.k8s.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-088ae15ae985c4286],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-088ae15ae985c4286,DevicePath:,},},Config:nil,},}
Jan 28 13:18:01.895: INFO: 
Logging kubelet events for node i-00e83413dfa35ee6b
Jan 28 13:18:02.112: INFO: 
Logging pods the kubelet thinks is on node i-00e83413dfa35ee6b
Jan 28 13:18:02.342: INFO: webserver-54977698-ql29p started at 2023-01-28 13:13:08 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Container httpd ready: true, restart count 0
Jan 28 13:18:02.342: INFO: service-proxy-disabled-dt7t6 started at 2023-01-28 13:13:59 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Container service-proxy-disabled ready: true, restart count 0
Jan 28 13:18:02.342: INFO: cilium-p9r7t started at 2023-01-28 12:52:28 +0000 UTC (1+1 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 28 13:18:02.342: INFO: 	Container cilium-agent ready: true, restart count 3
Jan 28 13:18:02.342: INFO: pod-subpath-test-preprovisionedpv-htrf started at 2023-01-28 13:18:01 +0000 UTC (2+1 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Init container init-volume-preprovisionedpv-htrf ready: false, restart count 0
Jan 28 13:18:02.342: INFO: 	Init container test-init-subpath-preprovisionedpv-htrf ready: false, restart count 0
Jan 28 13:18:02.342: INFO: 	Container test-container-subpath-preprovisionedpv-htrf ready: false, restart count 0
Jan 28 13:18:02.342: INFO: csi-mockplugin-0 started at 2023-01-28 13:17:58 +0000 UTC (0+3 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 28 13:18:02.342: INFO: 	Container driver-registrar ready: true, restart count 0
Jan 28 13:18:02.342: INFO: 	Container mock ready: true, restart count 0
Jan 28 13:18:02.342: INFO: netserver-0 started at 2023-01-28 13:12:48 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:02.342: INFO: csi-mockplugin-0 started at 2023-01-28 13:16:53 +0000 UTC (0+4 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Container busybox ready: true, restart count 0
Jan 28 13:18:02.342: INFO: 	Container csi-provisioner ready: true, restart count 2
Jan 28 13:18:02.342: INFO: 	Container driver-registrar ready: true, restart count 0
Jan 28 13:18:02.342: INFO: 	Container mock ready: true, restart count 0
Jan 28 13:18:02.342: INFO: netserver-0 started at 2023-01-28 13:14:01 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:02.342: INFO: pod-subpath-test-preprovisionedpv-wmm7 started at 2023-01-28 13:17:46 +0000 UTC (1+1 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Init container init-volume-preprovisionedpv-wmm7 ready: true, restart count 0
Jan 28 13:18:02.342: INFO: 	Container test-container-subpath-preprovisionedpv-wmm7 ready: true, restart count 0
Jan 28 13:18:02.342: INFO: ss2-2 started at 2023-01-28 13:13:22 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:02.342: INFO: pod-client started at 2023-01-28 13:13:31 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Container pod-client ready: true, restart count 0
Jan 28 13:18:02.342: INFO: ebs-csi-node-qfplb started at 2023-01-28 12:52:28 +0000 UTC (0+3 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 28 13:18:02.342: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 28 13:18:02.342: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 28 13:18:02.342: INFO: ss-1 started at 2023-01-28 13:17:10 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:02.342: INFO: webserver-54977698-5pcd4 started at 2023-01-28 13:12:49 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Container httpd ready: true, restart count 0
Jan 28 13:18:02.342: INFO: hostexec-i-00e83413dfa35ee6b-59mg8 started at 2023-01-28 13:17:30 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 28 13:18:02.342: INFO: hostexec-i-00e83413dfa35ee6b-hbqnp started at 2023-01-28 13:17:47 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:02.342: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 28 13:18:03.110: INFO: 
Latency metrics for node i-00e83413dfa35ee6b
Jan 28 13:18:03.110: INFO: 
Logging node info for node i-011ed997f238bea92
Jan 28 13:18:03.325: INFO: Node Info: &Node{ObjectMeta:{i-011ed997f238bea92    b3924a8c-3c07-4e89-9655-6dede0550248 43562 0 2023-01-28 12:52:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-011ed997f238bea92 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-011ed997f238bea92 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-5559":"i-011ed997f238bea92","csi-mock-csi-mock-volumes-8888":"csi-mock-csi-mock-volumes-8888","ebs.csi.aws.com":"i-011ed997f238bea92"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 12:52:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 12:52:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 12:52:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 12:52:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 12:52:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-01-28 13:15:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-01-28 13:16:05 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-011ed997f238bea92,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 13:15:57 +0000 UTC,LastTransitionTime:2023-01-28 12:52:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 13:15:57 +0000 UTC,LastTransitionTime:2023-01-28 12:52:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 13:15:57 +0000 UTC,LastTransitionTime:2023-01-28 12:52:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 13:15:57 +0000 UTC,LastTransitionTime:2023-01-28 12:52:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.54,},NodeAddress{Type:ExternalIP,Address:54.254.241.213,},NodeAddress{Type:InternalDNS,Address:i-011ed997f238bea92.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-011ed997f238bea92.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-241-213.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2847264721fc60b1e2b62ff5f906af,SystemUUID:ec284726-4721-fc60-b1e2-b62ff5f906af,BootID:c04be1bb-4aec-4103-8d4e-d3b345c3abb6,KernelVersion:5.15.0-1028-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.25.6,KubeProxyVersion:v1.25.6,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.6],SizeBytes:63291593,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-5559^3feeda26-9f0d-11ed-84c3-66b7a3d0f20d kubernetes.io/csi/ebs.csi.aws.com^vol-00e9c4b4c411a4354],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-00e9c4b4c411a4354,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5559^3feeda26-9f0d-11ed-84c3-66b7a3d0f20d,DevicePath:,},},Config:nil,},}
Jan 28 13:18:03.325: INFO: 
Logging kubelet events for node i-011ed997f238bea92
Jan 28 13:18:03.542: INFO: 
Logging pods the kubelet thinks is on node i-011ed997f238bea92
Jan 28 13:18:03.767: INFO: netserver-1 started at 2023-01-28 13:12:48 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.767: INFO: 	Container webserver ready: false, restart count 0
Jan 28 13:18:03.767: INFO: webserver-54977698-4sqcm started at 2023-01-28 13:13:07 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.767: INFO: 	Container httpd ready: false, restart count 0
Jan 28 13:18:03.768: INFO: pod-c161f54c-125e-4395-be49-581387fc94ee started at 2023-01-28 13:13:26 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container write-pod ready: false, restart count 0
Jan 28 13:18:03.768: INFO: exec-volume-test-preprovisionedpv-qf68 started at 2023-01-28 13:13:46 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container exec-container-preprovisionedpv-qf68 ready: false, restart count 0
Jan 28 13:18:03.768: INFO: csi-hostpathplugin-0 started at 2023-01-28 13:14:05 +0000 UTC (0+7 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container csi-attacher ready: false, restart count 0
Jan 28 13:18:03.768: INFO: 	Container csi-provisioner ready: false, restart count 0
Jan 28 13:18:03.768: INFO: 	Container csi-resizer ready: false, restart count 0
Jan 28 13:18:03.768: INFO: 	Container csi-snapshotter ready: false, restart count 0
Jan 28 13:18:03.768: INFO: 	Container hostpath ready: false, restart count 0
Jan 28 13:18:03.768: INFO: 	Container liveness-probe ready: false, restart count 0
Jan 28 13:18:03.768: INFO: 	Container node-driver-registrar ready: false, restart count 0
Jan 28 13:18:03.768: INFO: pod-subpath-test-preprovisionedpv-mnph started at 2023-01-28 13:16:47 +0000 UTC (1+2 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Init container test-init-subpath-preprovisionedpv-mnph ready: false, restart count 0
Jan 28 13:18:03.768: INFO: 	Container test-container-subpath-preprovisionedpv-mnph ready: false, restart count 0
Jan 28 13:18:03.768: INFO: 	Container test-container-volume-preprovisionedpv-mnph ready: false, restart count 0
Jan 28 13:18:03.768: INFO: csi-hostpathplugin-0 started at 2023-01-28 13:11:15 +0000 UTC (0+7 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 28 13:18:03.768: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 28 13:18:03.768: INFO: 	Container csi-resizer ready: true, restart count 0
Jan 28 13:18:03.768: INFO: 	Container csi-snapshotter ready: true, restart count 0
Jan 28 13:18:03.768: INFO: 	Container hostpath ready: true, restart count 0
Jan 28 13:18:03.768: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 28 13:18:03.768: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 28 13:18:03.768: INFO: service-proxy-disabled-rvj8q started at 2023-01-28 13:13:59 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container service-proxy-disabled ready: false, restart count 0
Jan 28 13:18:03.768: INFO: hostpath-symlink-prep-provisioning-2881 started at 2023-01-28 13:13:05 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container init-volume-provisioning-2881 ready: false, restart count 0
Jan 28 13:18:03.768: INFO: ss-1 started at 2023-01-28 13:15:53 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container webserver ready: false, restart count 0
Jan 28 13:18:03.768: INFO: hostexec-i-011ed997f238bea92-st2h8 started at 2023-01-28 13:13:30 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 28 13:18:03.768: INFO: hostexec-i-011ed997f238bea92-6ktg7 started at 2023-01-28 13:16:34 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 28 13:18:03.768: INFO: cilium-4dcmk started at 2023-01-28 12:52:36 +0000 UTC (1+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 28 13:18:03.768: INFO: 	Container cilium-agent ready: true, restart count 6
Jan 28 13:18:03.768: INFO: csi-mockplugin-0 started at 2023-01-28 13:12:29 +0000 UTC (0+4 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container busybox ready: true, restart count 0
Jan 28 13:18:03.768: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 28 13:18:03.768: INFO: 	Container driver-registrar ready: true, restart count 0
Jan 28 13:18:03.768: INFO: 	Container mock ready: true, restart count 0
Jan 28 13:18:03.768: INFO: inline-volume-tester-lgctp started at 2023-01-28 13:11:19 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container csi-volume-tester ready: false, restart count 0
Jan 28 13:18:03.768: INFO: coredns-85d58b74c8-9rvwc started at 2023-01-28 12:53:09 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container coredns ready: true, restart count 0
Jan 28 13:18:03.768: INFO: netserver-1 started at 2023-01-28 13:14:01 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container webserver ready: false, restart count 0
Jan 28 13:18:03.768: INFO: hostexec-i-011ed997f238bea92-fb4sv started at 2023-01-28 13:14:25 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 28 13:18:03.768: INFO: hostexec-i-011ed997f238bea92-hxw5m started at 2023-01-28 13:13:20 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 28 13:18:03.768: INFO: ss2-1 started at 2023-01-28 13:12:28 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container webserver ready: false, restart count 0
Jan 28 13:18:03.768: INFO: pod-server-1 started at 2023-01-28 13:13:36 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container agnhost-container ready: false, restart count 0
Jan 28 13:18:03.768: INFO: pod-subpath-test-preprovisionedpv-hkbx started at 2023-01-28 13:14:33 +0000 UTC (2+1 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Init container init-volume-preprovisionedpv-hkbx ready: false, restart count 0
Jan 28 13:18:03.768: INFO: 	Init container test-init-volume-preprovisionedpv-hkbx ready: false, restart count 0
Jan 28 13:18:03.768: INFO: 	Container test-container-subpath-preprovisionedpv-hkbx ready: false, restart count 0
Jan 28 13:18:03.768: INFO: ebs-csi-node-8wh45 started at 2023-01-28 12:52:36 +0000 UTC (0+3 container statuses recorded)
Jan 28 13:18:03.768: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 28 13:18:03.768: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 28 13:18:03.768: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 28 13:18:04.593: INFO: 
Latency metrics for node i-011ed997f238bea92
Jan 28 13:18:04.593: INFO: 
Logging node info for node i-04736cc52ac74b9d6
Jan 28 13:18:04.808: INFO: Node Info: &Node{ObjectMeta:{i-04736cc52ac74b9d6    f886f5d3-f71b-465b-8bf8-d2ad6142224c 45452 0 2023-01-28 12:52:30 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-04736cc52ac74b9d6 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-04736cc52ac74b9d6 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-04736cc52ac74b9d6"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 12:52:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 12:52:30 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 12:52:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-01-28 12:52:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-01-28 12:53:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 13:17:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-01-28 13:17:32 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-04736cc52ac74b9d6,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:31 +0000 UTC,LastTransitionTime:2023-01-28 12:52:30 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:31 +0000 UTC,LastTransitionTime:2023-01-28 12:52:30 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:31 +0000 UTC,LastTransitionTime:2023-01-28 12:52:30 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 13:17:31 +0000 UTC,LastTransitionTime:2023-01-28 12:53:11 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.85,},NodeAddress{Type:ExternalIP,Address:13.212.207.67,},NodeAddress{Type:InternalDNS,Address:i-04736cc52ac74b9d6.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-04736cc52ac74b9d6.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-212-207-67.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec257e4128091cdda716cfd4093ff7c9,SystemUUID:ec257e41-2809-1cdd-a716-cfd4093ff7c9,BootID:86228385-d1a8-45e0-a965-764379f12219,KernelVersion:5.15.0-1028-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.25.6,KubeProxyVersion:v1.25.6,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.6],SizeBytes:63291593,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c registry.k8s.io/e2e-test-images/nautilus:1.5],SizeBytes:49642095,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f registry.k8s.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-091a5c323546c884f kubernetes.io/csi/ebs.csi.aws.com^vol-0d582fe7c4d5e7b98],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-091a5c323546c884f,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0d582fe7c4d5e7b98,DevicePath:,},},Config:nil,},}
Jan 28 13:18:04.808: INFO: 
Logging kubelet events for node i-04736cc52ac74b9d6
Jan 28 13:18:05.025: INFO: 
Logging pods the kubelet thinks is on node i-04736cc52ac74b9d6
Jan 28 13:18:05.245: INFO: service-proxy-disabled-p2pjj started at 2023-01-28 13:13:59 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:05.245: INFO: 	Container service-proxy-disabled ready: true, restart count 0
Jan 28 13:18:05.245: INFO: ss-0 started at 2023-01-28 13:16:45 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:05.245: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:05.245: INFO: webserver-54977698-ksdtb started at 2023-01-28 13:12:36 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:05.245: INFO: 	Container httpd ready: true, restart count 0
Jan 28 13:18:05.245: INFO: ss2-0 started at 2023-01-28 13:12:42 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:05.245: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:05.245: INFO: cilium-6lzg7 started at 2023-01-28 12:52:31 +0000 UTC (1+1 container statuses recorded)
Jan 28 13:18:05.245: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 28 13:18:05.245: INFO: 	Container cilium-agent ready: true, restart count 3
Jan 28 13:18:05.245: INFO: ebs-csi-node-s9nrk started at 2023-01-28 12:52:31 +0000 UTC (0+3 container statuses recorded)
Jan 28 13:18:05.245: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 28 13:18:05.245: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 28 13:18:05.245: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 28 13:18:05.245: INFO: ss-0 started at 2023-01-28 13:15:07 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:05.245: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:05.245: INFO: netserver-2 started at 2023-01-28 13:14:01 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:05.245: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:05.245: INFO: liveness-c7ea544d-b26a-4808-9bc3-4057a98e77f1 started at 2023-01-28 13:17:10 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:05.245: INFO: 	Container agnhost-container ready: true, restart count 2
Jan 28 13:18:05.245: INFO: netserver-2 started at 2023-01-28 13:12:48 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:05.245: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:05.993: INFO: 
Latency metrics for node i-04736cc52ac74b9d6
Jan 28 13:18:05.993: INFO: 
Logging node info for node i-08256d641436a507d
Jan 28 13:18:06.208: INFO: Node Info: &Node{ObjectMeta:{i-08256d641436a507d    72e91885-e899-4272-9caa-03bda484a486 39774 0 2023-01-28 12:50:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-08256d641436a507d kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-08256d641436a507d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 12:50:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-01-28 12:50:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 12:51:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 12:51:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-01-28 12:51:19 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-28 13:13:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-08256d641436a507d,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892264960 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787407360 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 13:13:25 +0000 UTC,LastTransitionTime:2023-01-28 12:50:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 13:13:25 +0000 UTC,LastTransitionTime:2023-01-28 12:50:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 13:13:25 +0000 UTC,LastTransitionTime:2023-01-28 12:50:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 13:13:25 +0000 UTC,LastTransitionTime:2023-01-28 12:51:19 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.46.131,},NodeAddress{Type:ExternalIP,Address:54.255.178.176,},NodeAddress{Type:InternalDNS,Address:i-08256d641436a507d.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-08256d641436a507d.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-255-178-176.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec23fc2ff8b73bac68ca7ff21819b359,SystemUUID:ec23fc2f-f8b7-3bac-68ca-7ff21819b359,BootID:a83d4d4c-9cad-4b57-b033-8eeba43e9638,KernelVersion:5.15.0-1028-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.25.6,KubeProxyVersion:v1.25.6,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:231502799,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.25.6],SizeBytes:129108435,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.25.6],SizeBytes:118450489,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.6],SizeBytes:63291593,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.25.6],SizeBytes:51931448,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.2],SizeBytes:43830714,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.2],SizeBytes:42821730,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:26802430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:dcccdfba225e93ba2060a4c0b9072b50b0a564354c37bba6ed3ce89c326db58c registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.2],SizeBytes:18280697,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.2],SizeBytes:4965809,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 28 13:18:06.208: INFO: 
Logging kubelet events for node i-08256d641436a507d
Jan 28 13:18:06.425: INFO: 
Logging pods the kubelet thinks is on node i-08256d641436a507d
Jan 28 13:18:06.648: INFO: etcd-manager-main-i-08256d641436a507d started at 2023-01-28 12:49:52 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:06.649: INFO: 	Container etcd-manager ready: true, restart count 0
Jan 28 13:18:06.649: INFO: kube-controller-manager-i-08256d641436a507d started at 2023-01-28 12:49:52 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:06.649: INFO: 	Container kube-controller-manager ready: true, restart count 2
Jan 28 13:18:06.649: INFO: kube-scheduler-i-08256d641436a507d started at 2023-01-28 12:49:52 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:06.649: INFO: 	Container kube-scheduler ready: true, restart count 0
Jan 28 13:18:06.649: INFO: ebs-csi-controller-749b596fb7-pltks started at 2023-01-28 12:50:58 +0000 UTC (0+5 container statuses recorded)
Jan 28 13:18:06.649: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 28 13:18:06.649: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 28 13:18:06.649: INFO: 	Container csi-resizer ready: true, restart count 0
Jan 28 13:18:06.649: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 28 13:18:06.649: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 28 13:18:06.649: INFO: aws-cloud-controller-manager-qsphx started at 2023-01-28 12:50:58 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:06.649: INFO: 	Container aws-cloud-controller-manager ready: true, restart count 0
Jan 28 13:18:06.649: INFO: cilium-operator-674f5d6c65-x68tn started at 2023-01-28 12:51:00 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:06.649: INFO: 	Container cilium-operator ready: true, restart count 0
Jan 28 13:18:06.649: INFO: etcd-manager-events-i-08256d641436a507d started at 2023-01-28 12:49:52 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:06.649: INFO: 	Container etcd-manager ready: true, restart count 0
Jan 28 13:18:06.649: INFO: kube-apiserver-i-08256d641436a507d started at 2023-01-28 12:49:52 +0000 UTC (0+2 container statuses recorded)
Jan 28 13:18:06.649: INFO: 	Container healthcheck ready: true, restart count 0
Jan 28 13:18:06.649: INFO: 	Container kube-apiserver ready: true, restart count 1
Jan 28 13:18:06.649: INFO: etcd-manager-cilium-i-08256d641436a507d started at 2023-01-28 12:49:52 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:06.649: INFO: 	Container etcd-manager ready: true, restart count 0
Jan 28 13:18:06.649: INFO: ebs-csi-node-bj55j started at 2023-01-28 12:50:48 +0000 UTC (0+3 container statuses recorded)
Jan 28 13:18:06.649: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 28 13:18:06.649: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 28 13:18:06.649: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 28 13:18:06.649: INFO: cilium-sglkk started at 2023-01-28 12:50:58 +0000 UTC (1+1 container statuses recorded)
Jan 28 13:18:06.649: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 28 13:18:06.649: INFO: 	Container cilium-agent ready: true, restart count 1
Jan 28 13:18:06.649: INFO: dns-controller-56d4f686f6-5xr8q started at 2023-01-28 12:50:58 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:06.649: INFO: 	Container dns-controller ready: true, restart count 0
Jan 28 13:18:06.649: INFO: kops-controller-hh62m started at 2023-01-28 12:50:58 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:06.649: INFO: 	Container kops-controller ready: true, restart count 0
Jan 28 13:18:07.400: INFO: 
Latency metrics for node i-08256d641436a507d
Jan 28 13:18:07.400: INFO: 
Logging node info for node i-0cceed973097eb227
Jan 28 13:18:07.620: INFO: Node Info: &Node{ObjectMeta:{i-0cceed973097eb227    0d4efd12-5177-4438-95a1-cf4f7bc47cb0 45546 0 2023-01-28 12:52:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0cceed973097eb227 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0cceed973097eb227 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0cceed973097eb227"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 12:52:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 12:52:28 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 12:52:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 12:52:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 12:52:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-01-28 13:17:34 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 13:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0cceed973097eb227,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:39 +0000 UTC,LastTransitionTime:2023-01-28 12:52:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:39 +0000 UTC,LastTransitionTime:2023-01-28 12:52:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:39 +0000 UTC,LastTransitionTime:2023-01-28 12:52:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 13:17:39 +0000 UTC,LastTransitionTime:2023-01-28 12:52:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.50.150,},NodeAddress{Type:ExternalIP,Address:52.77.228.237,},NodeAddress{Type:InternalDNS,Address:i-0cceed973097eb227.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0cceed973097eb227.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-77-228-237.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bfb64b99756deba115b4f9f6a58b6,SystemUUID:ec2bfb64-b997-56de-ba11-5b4f9f6a58b6,BootID:104ee6bd-b21a-45d3-9eeb-f69e1fedb453,KernelVersion:5.15.0-1028-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.25.6,KubeProxyVersion:v1.25.6,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.6],SizeBytes:63291593,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-02b16e7f805809f6d kubernetes.io/csi/ebs.csi.aws.com^vol-0d793e6008e647344],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0d793e6008e647344,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02b16e7f805809f6d,DevicePath:,},},Config:nil,},}
Jan 28 13:18:07.620: INFO: 
Logging kubelet events for node i-0cceed973097eb227
Jan 28 13:18:07.837: INFO: 
Logging pods the kubelet thinks is on node i-0cceed973097eb227
Jan 28 13:18:08.057: INFO: webserver-54977698-hmpbz started at 2023-01-28 13:12:42 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:08.057: INFO: 	Container httpd ready: true, restart count 0
Jan 28 13:18:08.057: INFO: webserver-54977698-zgr7q started at 2023-01-28 13:12:54 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:08.057: INFO: 	Container httpd ready: true, restart count 0
Jan 28 13:18:08.057: INFO: coredns-autoscaler-5b9dc8bb99-t6qm5 started at 2023-01-28 12:52:48 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:08.057: INFO: 	Container autoscaler ready: true, restart count 0
Jan 28 13:18:08.057: INFO: ebs-csi-node-m7vpr started at 2023-01-28 12:52:29 +0000 UTC (0+3 container statuses recorded)
Jan 28 13:18:08.057: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 28 13:18:08.057: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 28 13:18:08.057: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 28 13:18:08.057: INFO: netserver-3 started at 2023-01-28 13:14:02 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:08.057: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:08.057: INFO: netserver-3 started at 2023-01-28 13:12:48 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:08.057: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:08.057: INFO: ss-2 started at 2023-01-28 13:15:35 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:08.057: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:08.057: INFO: pod-subpath-test-secret-gdwk started at 2023-01-28 13:17:58 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:08.057: INFO: 	Container test-container-subpath-secret-gdwk ready: true, restart count 0
Jan 28 13:18:08.057: INFO: cilium-j76rx started at 2023-01-28 12:52:29 +0000 UTC (1+1 container statuses recorded)
Jan 28 13:18:08.057: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 28 13:18:08.057: INFO: 	Container cilium-agent ready: true, restart count 3
Jan 28 13:18:08.057: INFO: ss-2 started at 2023-01-28 13:17:31 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:08.057: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:08.057: INFO: ss2-1 started at 2023-01-28 13:18:05 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:08.057: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:08.057: INFO: coredns-85d58b74c8-65c7c started at 2023-01-28 12:52:48 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:08.057: INFO: 	Container coredns ready: true, restart count 0
Jan 28 13:18:08.822: INFO: 
Latency metrics for node i-0cceed973097eb227
Jan 28 13:18:08.822: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
�[1mSTEP:�[0m Destroying namespace "nettest-4274" for this suite. �[38;5;243m01/28/23 13:18:09.038�[0m

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

Kubernetes e2e suite [It] [sig-storage] CSI mock volume storage capacity exhausted, late binding, with topology 6m53s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-storage\]\sCSI\smock\svolume\sstorage\scapacity\sexhausted\,\slate\sbinding\,\swith\stopology$'

test/e2e/storage/csi_mock_volume.go:1211
k8s.io/kubernetes/test/e2e/storage.glob..func2.14.1()
	test/e2e/storage/csi_mock_volume.go:1211 +0x626
				{"msg":"FAILED [sig-storage] CSI mock volume storage capacity exhausted, late binding, with topology","completed":26,"skipped":271,"failed":1,"failures":["[sig-storage] CSI mock volume storage capacity exhausted, late binding, with topology"]}

[BeforeEach] [sig-storage] CSI mock volume
  test/e2e/framework/framework.go:186
�[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m01/28/23 13:12:19.727�[0m
Jan 28 13:12:19.727: INFO: >>> kubeConfig: /root/.kube/config
�[1mSTEP:�[0m Building a namespace api object, basename csi-mock-volumes �[38;5;243m01/28/23 13:12:19.728�[0m
�[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m01/28/23 13:12:20.374�[0m
�[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m01/28/23 13:12:20.802�[0m
[It] exhausted, late binding, with topology
  test/e2e/storage/csi_mock_volume.go:1158
�[1mSTEP:�[0m Building a driver namespace object, basename csi-mock-volumes-8888 �[38;5;243m01/28/23 13:12:21.238�[0m
�[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m01/28/23 13:12:21.883�[0m
�[1mSTEP:�[0m deploying csi mock proxy �[38;5;243m01/28/23 13:12:22.312�[0m
Jan 28 13:12:23.177: INFO: creating *v1.ServiceAccount: csi-mock-volumes-8888-4534/csi-attacher
Jan 28 13:12:23.395: INFO: creating *v1.ClusterRole: external-attacher-runner-csi-mock-volumes-8888
Jan 28 13:12:23.395: INFO: Define cluster role external-attacher-runner-csi-mock-volumes-8888
Jan 28 13:12:23.614: INFO: creating *v1.ClusterRoleBinding: csi-attacher-role-csi-mock-volumes-8888
Jan 28 13:12:23.835: INFO: creating *v1.Role: csi-mock-volumes-8888-4534/external-attacher-cfg-csi-mock-volumes-8888
Jan 28 13:12:24.052: INFO: creating *v1.RoleBinding: csi-mock-volumes-8888-4534/csi-attacher-role-cfg
Jan 28 13:12:24.267: INFO: creating *v1.ServiceAccount: csi-mock-volumes-8888-4534/csi-provisioner
Jan 28 13:12:24.483: INFO: creating *v1.ClusterRole: external-provisioner-runner-csi-mock-volumes-8888
Jan 28 13:12:24.483: INFO: Define cluster role external-provisioner-runner-csi-mock-volumes-8888
Jan 28 13:12:24.701: INFO: creating *v1.ClusterRoleBinding: csi-provisioner-role-csi-mock-volumes-8888
Jan 28 13:12:24.919: INFO: creating *v1.Role: csi-mock-volumes-8888-4534/external-provisioner-cfg-csi-mock-volumes-8888
Jan 28 13:12:25.138: INFO: creating *v1.RoleBinding: csi-mock-volumes-8888-4534/csi-provisioner-role-cfg
Jan 28 13:12:25.354: INFO: creating *v1.ServiceAccount: csi-mock-volumes-8888-4534/csi-resizer
Jan 28 13:12:25.571: INFO: creating *v1.ClusterRole: external-resizer-runner-csi-mock-volumes-8888
Jan 28 13:12:25.571: INFO: Define cluster role external-resizer-runner-csi-mock-volumes-8888
Jan 28 13:12:25.790: INFO: creating *v1.ClusterRoleBinding: csi-resizer-role-csi-mock-volumes-8888
Jan 28 13:12:26.006: INFO: creating *v1.Role: csi-mock-volumes-8888-4534/external-resizer-cfg-csi-mock-volumes-8888
Jan 28 13:12:26.221: INFO: creating *v1.RoleBinding: csi-mock-volumes-8888-4534/csi-resizer-role-cfg
Jan 28 13:12:26.437: INFO: creating *v1.ServiceAccount: csi-mock-volumes-8888-4534/csi-snapshotter
Jan 28 13:12:26.653: INFO: creating *v1.ClusterRole: external-snapshotter-runner-csi-mock-volumes-8888
Jan 28 13:12:26.653: INFO: Define cluster role external-snapshotter-runner-csi-mock-volumes-8888
Jan 28 13:12:26.878: INFO: creating *v1.ClusterRoleBinding: csi-snapshotter-role-csi-mock-volumes-8888
Jan 28 13:12:27.094: INFO: creating *v1.Role: csi-mock-volumes-8888-4534/external-snapshotter-leaderelection-csi-mock-volumes-8888
Jan 28 13:12:27.310: INFO: creating *v1.RoleBinding: csi-mock-volumes-8888-4534/external-snapshotter-leaderelection
Jan 28 13:12:27.526: INFO: creating *v1.ServiceAccount: csi-mock-volumes-8888-4534/csi-mock
Jan 28 13:12:27.743: INFO: creating *v1.ClusterRoleBinding: csi-controller-attacher-role-csi-mock-volumes-8888
Jan 28 13:12:27.958: INFO: creating *v1.ClusterRoleBinding: csi-controller-provisioner-role-csi-mock-volumes-8888
Jan 28 13:12:28.174: INFO: creating *v1.ClusterRoleBinding: csi-controller-cluster-driver-registrar-role-csi-mock-volumes-8888
Jan 28 13:12:28.390: INFO: creating *v1.ClusterRoleBinding: psp-csi-controller-driver-registrar-role-csi-mock-volumes-8888
Jan 28 13:12:28.606: INFO: creating *v1.ClusterRoleBinding: csi-controller-resizer-role-csi-mock-volumes-8888
Jan 28 13:12:28.822: INFO: creating *v1.ClusterRoleBinding: csi-controller-snapshotter-role-csi-mock-volumes-8888
Jan 28 13:12:29.038: INFO: creating *v1.StorageClass: csi-mock-sc-csi-mock-volumes-8888
Jan 28 13:12:29.256: INFO: creating *v1.StatefulSet: csi-mock-volumes-8888-4534/csi-mockplugin
Jan 28 13:12:29.474: INFO: creating *v1.CSIDriver: csi-mock-csi-mock-volumes-8888
Jan 28 13:12:29.690: INFO: waiting up to 4m0s for CSIDriver "csi-mock-csi-mock-volumes-8888"
Jan 28 13:12:29.906: INFO: waiting for CSIDriver csi-mock-csi-mock-volumes-8888 to register on node i-011ed997f238bea92
I0128 13:12:38.256440    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Identity/Probe","Request":{},"Response":{"ready":{"value":true}},"Error":"","FullError":null}
I0128 13:12:38.472365    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Identity/GetPluginInfo","Request":{},"Response":{"name":"csi-mock-csi-mock-volumes-8888","vendor_version":"0.3.0","manifest":{"url":"https://github.com/kubernetes/kubernetes/tree/master/test/e2e/storage/drivers/csi-test/mock"}},"Error":"","FullError":null}
I0128 13:12:38.692903    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Identity/GetPluginCapabilities","Request":{},"Response":{"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"VolumeExpansion":{"type":1}}},{"Type":{"Service":{"type":2}}}]},"Error":"","FullError":null}
I0128 13:12:38.907798    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Controller/ControllerGetCapabilities","Request":{},"Response":{"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":10}}},{"Type":{"Rpc":{"type":4}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":8}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":12}}},{"Type":{"Rpc":{"type":11}}},{"Type":{"Rpc":{"type":9}}}]},"Error":"","FullError":null}
I0128 13:12:39.361101    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Identity/GetPluginInfo","Request":{},"Response":{"name":"csi-mock-csi-mock-volumes-8888","vendor_version":"0.3.0","manifest":{"url":"https://github.com/kubernetes/kubernetes/tree/master/test/e2e/storage/drivers/csi-test/mock"}},"Error":"","FullError":null}
I0128 13:12:40.470071    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeGetInfo","Request":{},"Response":{"node_id":"csi-mock-csi-mock-volumes-8888","accessible_topology":{"segments":{"io.kubernetes.storage.mock/node":"some-mock-node"}}},"Error":"","FullError":null}
�[1mSTEP:�[0m Creating pod �[38;5;243m01/28/23 13:12:47.452�[0m
Jan 28 13:12:47.669: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Jan 28 13:12:48.104: INFO: Waiting up to 5m0s for pod "pvc-volume-tester-z4mbj" in namespace "csi-mock-volumes-8888" to be "running"
I0128 13:12:48.121005    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Controller/CreateVolume","Request":{"name":"pvc-79c1692e-f2a8-4e5a-94d7-e580686588ac","capacity_range":{"required_bytes":1073741824},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}],"accessibility_requirements":{"requisite":[{"segments":{"io.kubernetes.storage.mock/node":"some-mock-node"}}],"preferred":[{"segments":{"io.kubernetes.storage.mock/node":"some-mock-node"}}]}},"Response":null,"Error":"rpc error: code = ResourceExhausted desc = fake error","FullError":{"code":8,"message":"fake error"}}
Jan 28 13:12:48.319: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 215.015833ms
I0128 13:12:50.392996    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Controller/CreateVolume","Request":{"name":"pvc-79c1692e-f2a8-4e5a-94d7-e580686588ac","capacity_range":{"required_bytes":1073741824},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}],"accessibility_requirements":{"requisite":[{"segments":{"io.kubernetes.storage.mock/node":"some-mock-node"}}],"preferred":[{"segments":{"io.kubernetes.storage.mock/node":"some-mock-node"}}]}},"Response":{"volume":{"capacity_bytes":1073741824,"volume_id":"4","volume_context":{"name":"pvc-79c1692e-f2a8-4e5a-94d7-e580686588ac"},"accessible_topology":[{"segments":{"io.kubernetes.storage.mock/node":"some-mock-node"}}]}},"Error":"","FullError":null}
Jan 28 13:12:50.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2.430167236s
I0128 13:12:51.518794    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeGetCapabilities","Request":{},"Response":{"capabilities":[{"Type":{"Rpc":{}}},{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":4}}}]},"Error":"","FullError":null}
I0128 13:12:51.734562    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeGetCapabilities","Request":{},"Response":{"capabilities":[{"Type":{"Rpc":{}}},{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":4}}}]},"Error":"","FullError":null}
I0128 13:12:51.953561    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeGetCapabilities","Request":{},"Response":{"capabilities":[{"Type":{"Rpc":{}}},{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":4}}}]},"Error":"","FullError":null}
Jan 28 13:12:52.169: INFO: >>> kubeConfig: /root/.kube/config
Jan 28 13:12:52.170: INFO: ExecWithOptions: Clientset creation
Jan 28 13:12:52.170: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-grid-cilium-etcd-u2204-k25-ko26.test-cncf-aws.k8s.io/api/v1/namespaces/csi-mock-volumes-8888-4534/pods/csi-mockplugin-0/exec?command=sh&command=-c&command=if+%21+%5B+-e+%27%2Fvar%2Flib%2Fkubelet%2Fplugins%2Fkubernetes.io%2Fcsi%2Fcsi-mock-csi-mock-volumes-8888%2F4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a%2Fglobalmount%27+%5D%3B+then+echo+notexist%3B+elif+%5B+-d+%27%2Fvar%2Flib%2Fkubelet%2Fplugins%2Fkubernetes.io%2Fcsi%2Fcsi-mock-csi-mock-volumes-8888%2F4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a%2Fglobalmount%27+%5D%3B+then+echo+dir%3B+else+echo+nodir%3B+fi&container=busybox&container=busybox&stderr=true&stdout=true)
Jan 28 13:12:52.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4.43082859s
I0128 13:12:53.547012    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeStageVolume","Request":{"volume_id":"4","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/csi-mock-csi-mock-volumes-8888/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"name":"pvc-79c1692e-f2a8-4e5a-94d7-e580686588ac","storage.kubernetes.io/csiProvisionerIdentity":"1674911559015-8081-csi-mock-csi-mock-volumes-8888"}},"Response":{},"Error":"","FullError":null}
I0128 13:12:53.765352    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeGetCapabilities","Request":{},"Response":{"capabilities":[{"Type":{"Rpc":{}}},{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":4}}}]},"Error":"","FullError":null}
I0128 13:12:53.982666    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeGetCapabilities","Request":{},"Response":{"capabilities":[{"Type":{"Rpc":{}}},{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":4}}}]},"Error":"","FullError":null}
I0128 13:12:54.198489    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeGetCapabilities","Request":{},"Response":{"capabilities":[{"Type":{"Rpc":{}}},{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":4}}}]},"Error":"","FullError":null}
Jan 28 13:12:54.413: INFO: >>> kubeConfig: /root/.kube/config
Jan 28 13:12:54.414: INFO: ExecWithOptions: Clientset creation
Jan 28 13:12:54.414: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-grid-cilium-etcd-u2204-k25-ko26.test-cncf-aws.k8s.io/api/v1/namespaces/csi-mock-volumes-8888-4534/pods/csi-mockplugin-0/exec?command=sh&command=-c&command=if+%21+%5B+-e+%27%2Fvar%2Flib%2Fkubelet%2Fpods%2F0818f735-5dda-41ba-8a33-f952e95834a3%2Fvolumes%2Fkubernetes.io~csi%2Fpvc-79c1692e-f2a8-4e5a-94d7-e580686588ac%2Fmount%27+%5D%3B+then+echo+notexist%3B+elif+%5B+-d+%27%2Fvar%2Flib%2Fkubelet%2Fpods%2F0818f735-5dda-41ba-8a33-f952e95834a3%2Fvolumes%2Fkubernetes.io~csi%2Fpvc-79c1692e-f2a8-4e5a-94d7-e580686588ac%2Fmount%27+%5D%3B+then+echo+dir%3B+else+echo+nodir%3B+fi&container=busybox&container=busybox&stderr=true&stdout=true)
Jan 28 13:12:54.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 6.430129292s
Jan 28 13:12:55.788: INFO: >>> kubeConfig: /root/.kube/config
Jan 28 13:12:55.789: INFO: ExecWithOptions: Clientset creation
Jan 28 13:12:55.789: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-grid-cilium-etcd-u2204-k25-ko26.test-cncf-aws.k8s.io/api/v1/namespaces/csi-mock-volumes-8888-4534/pods/csi-mockplugin-0/exec?command=sh&command=-c&command=if+%21+%5B+-e+%27%2Fvar%2Flib%2Fkubelet%2Fpods%2F0818f735-5dda-41ba-8a33-f952e95834a3%2Fvolumes%2Fkubernetes.io~csi%2Fpvc-79c1692e-f2a8-4e5a-94d7-e580686588ac%2Fmount%27+%5D%3B+then+echo+notexist%3B+elif+%5B+-d+%27%2Fvar%2Flib%2Fkubelet%2Fpods%2F0818f735-5dda-41ba-8a33-f952e95834a3%2Fvolumes%2Fkubernetes.io~csi%2Fpvc-79c1692e-f2a8-4e5a-94d7-e580686588ac%2Fmount%27+%5D%3B+then+echo+dir%3B+else+echo+nodir%3B+fi&container=busybox&container=busybox&stderr=true&stdout=true)
Jan 28 13:12:56.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 8.430237589s
Jan 28 13:12:57.155: INFO: >>> kubeConfig: /root/.kube/config
Jan 28 13:12:57.156: INFO: ExecWithOptions: Clientset creation
Jan 28 13:12:57.156: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-grid-cilium-etcd-u2204-k25-ko26.test-cncf-aws.k8s.io/api/v1/namespaces/csi-mock-volumes-8888-4534/pods/csi-mockplugin-0/exec?command=mkdir&command=%2Fvar%2Flib%2Fkubelet%2Fpods%2F0818f735-5dda-41ba-8a33-f952e95834a3%2Fvolumes%2Fkubernetes.io~csi%2Fpvc-79c1692e-f2a8-4e5a-94d7-e580686588ac%2Fmount&container=busybox&container=busybox&stderr=true&stdout=true)
I0128 13:12:58.524928    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodePublishVolume","Request":{"volume_id":"4","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/csi-mock-csi-mock-volumes-8888/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a/globalmount","target_path":"/var/lib/kubelet/pods/0818f735-5dda-41ba-8a33-f952e95834a3/volumes/kubernetes.io~csi/pvc-79c1692e-f2a8-4e5a-94d7-e580686588ac/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"name":"pvc-79c1692e-f2a8-4e5a-94d7-e580686588ac","storage.kubernetes.io/csiProvisionerIdentity":"1674911559015-8081-csi-mock-csi-mock-volumes-8888"}},"Response":{},"Error":"","FullError":null}
Jan 28 13:12:58.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 10.43064394s
Jan 28 13:13:00.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 12.429840276s
Jan 28 13:13:02.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 14.430712448s
Jan 28 13:13:04.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 16.430819868s
Jan 28 13:13:06.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 18.430530195s
Jan 28 13:13:08.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 20.430820023s
Jan 28 13:13:10.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 22.43072784s
Jan 28 13:13:12.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 24.430770571s
Jan 28 13:13:14.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 26.430377656s
Jan 28 13:13:16.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 28.430114425s
Jan 28 13:13:18.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 30.429925616s
Jan 28 13:13:20.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 32.430695779s
Jan 28 13:13:22.536: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 34.431563041s
Jan 28 13:13:24.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 36.430036821s
Jan 28 13:13:26.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 38.430882607s
Jan 28 13:13:28.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 40.430627748s
Jan 28 13:13:30.536: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 42.432248753s
Jan 28 13:13:32.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 44.430437384s
Jan 28 13:13:34.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 46.429632847s
I0128 13:13:35.310589    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeGetCapabilities","Request":{},"Response":{"capabilities":[{"Type":{"Rpc":{}}},{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":4}}}]},"Error":"","FullError":null}
I0128 13:13:35.528265    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeGetVolumeStats","Request":{"volume_id":"4","volume_path":"/var/lib/kubelet/pods/0818f735-5dda-41ba-8a33-f952e95834a3/volumes/kubernetes.io~csi/pvc-79c1692e-f2a8-4e5a-94d7-e580686588ac/mount"},"Response":{"usage":[{"total":1073741824,"unit":1}],"volume_condition":{}},"Error":"","FullError":null}
Jan 28 13:13:36.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 48.429730165s
Jan 28 13:13:38.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 50.430731962s
Jan 28 13:13:40.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 52.430207275s
Jan 28 13:13:42.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 54.430070079s
Jan 28 13:13:44.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 56.430241879s
Jan 28 13:13:46.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 58.430393333s
Jan 28 13:13:48.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.430544812s
Jan 28 13:13:50.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.430774265s
Jan 28 13:13:52.536: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.432242978s
Jan 28 13:13:54.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.430021205s
Jan 28 13:13:56.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.431011685s
Jan 28 13:13:58.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.429947156s
Jan 28 13:14:00.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.429924804s
Jan 28 13:14:02.544: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.439667746s
Jan 28 13:14:04.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.431105235s
Jan 28 13:14:06.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.429898995s
Jan 28 13:14:08.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.430632925s
Jan 28 13:14:10.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.430719718s
Jan 28 13:14:12.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.43104724s
Jan 28 13:14:14.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.430914532s
Jan 28 13:14:16.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.430629249s
Jan 28 13:14:18.539: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.434833799s
Jan 28 13:14:20.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.431029446s
Jan 28 13:14:22.538: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.433987148s
Jan 28 13:14:24.537: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.432938613s
Jan 28 13:14:26.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m38.43047986s
Jan 28 13:14:28.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m40.429765392s
Jan 28 13:14:30.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m42.430711499s
Jan 28 13:14:32.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.430784739s
Jan 28 13:14:34.539: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.435057775s
Jan 28 13:14:36.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.430719489s
Jan 28 13:14:38.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m50.430856738s
Jan 28 13:14:40.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m52.429977344s
Jan 28 13:14:42.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m54.430388325s
Jan 28 13:14:44.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m56.430828888s
Jan 28 13:14:46.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m58.430695741s
Jan 28 13:14:48.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m0.430856789s
Jan 28 13:14:50.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m2.430496948s
Jan 28 13:14:52.539: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m4.43496126s
Jan 28 13:14:54.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.430161682s
Jan 28 13:14:56.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.430140258s
Jan 28 13:14:58.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.430324432s
Jan 28 13:15:00.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.430771308s
Jan 28 13:15:02.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.430751408s
Jan 28 13:15:04.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.430376957s
Jan 28 13:15:06.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.430760466s
Jan 28 13:15:08.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.43038822s
Jan 28 13:15:10.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.429976428s
I0128 13:15:11.047052    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeGetCapabilities","Request":{},"Response":{"capabilities":[{"Type":{"Rpc":{}}},{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":4}}}]},"Error":"","FullError":null}
I0128 13:15:11.263718    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeGetVolumeStats","Request":{"volume_id":"4","volume_path":"/var/lib/kubelet/pods/0818f735-5dda-41ba-8a33-f952e95834a3/volumes/kubernetes.io~csi/pvc-79c1692e-f2a8-4e5a-94d7-e580686588ac/mount"},"Response":{"usage":[{"total":1073741824,"unit":1}],"volume_condition":{}},"Error":"","FullError":null}
Jan 28 13:15:12.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.430022762s
Jan 28 13:15:14.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.43029906s
Jan 28 13:15:16.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.429863728s
Jan 28 13:15:18.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.430270217s
Jan 28 13:15:20.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.430662255s
Jan 28 13:15:22.536: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.431381128s
Jan 28 13:15:24.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.430393532s
Jan 28 13:15:26.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.430713064s
Jan 28 13:15:28.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.430265409s
Jan 28 13:15:30.540: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.435577756s
Jan 28 13:15:32.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.430243275s
Jan 28 13:15:34.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.430783773s
Jan 28 13:15:36.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.430785173s
Jan 28 13:15:38.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.430637089s
Jan 28 13:15:40.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.430814369s
Jan 28 13:15:42.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.430784424s
Jan 28 13:15:44.536: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.431481025s
Jan 28 13:15:46.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 2m58.430199318s
Jan 28 13:15:48.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m0.430894769s
Jan 28 13:15:50.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m2.430275429s
Jan 28 13:15:52.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.430615418s
Jan 28 13:15:54.536: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m6.431673353s
Jan 28 13:15:56.537: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m8.432590511s
Jan 28 13:15:58.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.430294746s
Jan 28 13:16:00.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.430470807s
Jan 28 13:16:02.536: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m14.431608813s
Jan 28 13:16:04.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m16.431248924s
Jan 28 13:16:06.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m18.430285545s
Jan 28 13:16:08.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m20.430306475s
Jan 28 13:16:10.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m22.430288187s
Jan 28 13:16:12.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.430815781s
Jan 28 13:16:14.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.430805198s
Jan 28 13:16:16.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.430125552s
Jan 28 13:16:18.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.430719512s
Jan 28 13:16:20.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.430711136s
Jan 28 13:16:22.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m34.430435946s
Jan 28 13:16:24.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m36.431190671s
Jan 28 13:16:26.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m38.430584868s
Jan 28 13:16:28.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m40.431010242s
Jan 28 13:16:30.536: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m42.432065708s
Jan 28 13:16:32.537: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m44.432622191s
Jan 28 13:16:34.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m46.430210601s
Jan 28 13:16:36.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m48.429873957s
Jan 28 13:16:38.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m50.430618819s
Jan 28 13:16:40.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m52.430195985s
Jan 28 13:16:42.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m54.431141884s
Jan 28 13:16:44.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m56.430457815s
Jan 28 13:16:46.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 3m58.43027461s
Jan 28 13:16:48.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m0.430907183s
Jan 28 13:16:50.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m2.430378619s
Jan 28 13:16:52.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m4.431004678s
Jan 28 13:16:54.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.43046167s
Jan 28 13:16:56.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.430953934s
Jan 28 13:16:58.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.430514143s
Jan 28 13:17:00.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.430129921s
I0128 13:17:01.720029    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeGetCapabilities","Request":{},"Response":{"capabilities":[{"Type":{"Rpc":{}}},{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":4}}}]},"Error":"","FullError":null}
I0128 13:17:01.936570    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeGetVolumeStats","Request":{"volume_id":"4","volume_path":"/var/lib/kubelet/pods/0818f735-5dda-41ba-8a33-f952e95834a3/volumes/kubernetes.io~csi/pvc-79c1692e-f2a8-4e5a-94d7-e580686588ac/mount"},"Response":{"usage":[{"total":1073741824,"unit":1}],"volume_condition":{}},"Error":"","FullError":null}
Jan 28 13:17:02.540: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.435555691s
Jan 28 13:17:04.536: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m16.431419611s
Jan 28 13:17:06.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m18.43085541s
Jan 28 13:17:08.536: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m20.431814198s
Jan 28 13:17:10.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m22.430634885s
Jan 28 13:17:12.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m24.430333976s
Jan 28 13:17:14.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.430690295s
Jan 28 13:17:16.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.430445916s
Jan 28 13:17:18.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.430129972s
Jan 28 13:17:20.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.430514824s
Jan 28 13:17:22.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.430686942s
Jan 28 13:17:24.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.430467936s
Jan 28 13:17:26.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.43089747s
Jan 28 13:17:28.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.430511904s
Jan 28 13:17:30.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.43023091s
Jan 28 13:17:32.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.430628591s
Jan 28 13:17:34.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.430492881s
Jan 28 13:17:36.536: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.431896335s
Jan 28 13:17:38.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.430302681s
Jan 28 13:17:40.534: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.430247979s
Jan 28 13:17:42.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.430523727s
Jan 28 13:17:44.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.43062846s
Jan 28 13:17:46.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.431183599s
Jan 28 13:17:48.535: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.431037639s
Jan 28 13:17:48.751: INFO: Pod "pvc-volume-tester-z4mbj": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.646377971s
Jan 28 13:17:48.752: INFO: Unexpected error: failed to start pod: 
    <*pod.timeoutError | 0xc003a41260>: {
        msg: "timed out while waiting for pod csi-mock-volumes-8888/pvc-volume-tester-z4mbj to be running",
        observedObjects: [
            <*v1.Pod | 0xc001601400>{
                TypeMeta: {Kind: "", APIVersion: ""},
                ObjectMeta: {
                    Name: "pvc-volume-tester-z4mbj",
                    GenerateName: "pvc-volume-tester-",
                    Namespace: "csi-mock-volumes-8888",
                    SelfLink: "",
                    UID: "0818f735-5dda-41ba-8a33-f952e95834a3",
                    ResourceVersion: "38258",
                    Generation: 0,
                    CreationTimestamp: {
                        Time: {
                            wall: 0,
                            ext: 63810508367,
                            loc: {
                                name: "Local",
                                zone: [
                                    {name: "UTC", offset: 0, isDST: false},
                                ],
                                tx: [
                                    {
                                        when: -9223372036854775808,
                                        index: 0,
                                        isstd: false,
                                        isutc: false,
                                    },
                                ],
                                extend: "UTC0",
                                cacheStart: 9223372036854775807,
                                cacheEnd: 9223372036854775807,
                                cacheZone: {name: "UTC", offset: 0, isDST: false},
                            },
                        },
                    },
                    DeletionTimestamp: nil,
                    DeletionGracePeriodSeconds: nil,
                    Labels: nil,
                    Annotations: nil,
                    OwnerReferences: nil,
                    Finalizers: nil,
                    ManagedFields: [
                        {
                            Manager: "e2e.test",
                            Operation: "Update",
                            APIVersion: "v1",
                            Time: {
                                Time: {
                                    wall: 0,
                                    ext: 63810508367,
                                    loc: {
                                        name: "Local",
                                        zone: [...],
                                        tx: [...],
                                        extend: "UTC0",
                                        cacheStart: 9223372036854775807,
                                        cacheEnd: 9223372036854775807,
                                        cacheZone: {name: ..., offset: ..., isDST: ...},
                                    },
                                },
                            },
                            FieldsType: "FieldsV1",
                            FieldsV1: {
                                Raw: "{\"f:metadata\":{\"f:generateName\":{}},\"f:spec\":{\"f:affinity\":{\".\":{},\"f:nodeAffinity\":{\".\":{},\"f:requiredDuringSchedulingIgnoredDuringExecution\":{}}},\"f:containers\":{\"k:{\\\"name\\\":\\\"volume-tester\\\"}\":{\".\":{},\"f:image\":{},\"f:imagePullPolicy\":{},\"f:name\":{},\"f:resources\":{},\"f:terminationMessagePath\":{},\"f:terminationMessagePolicy\":{},\"f:volumeMounts\":{\".\":{},\"k:{\\\"mountPath\\\":\\\"/mnt/test\\\"}\":{\".\":{},\"f:mountPath\":{},\"f:name\":{}}}}},\"f:dnsPolicy\":{},\"f:enableServiceLinks\":{},\"f:restartPolicy\":{},\"f:schedulerName\":{},\"f:securityContext\":{},\"f:terminationGracePeriodSeconds\":{},\"f:volumes\":{\".\":{},\"k:{\\\"name\\\":\\\"my-volume\\\"}\":{\".\":{},\"f:name\":{},\"f:persistentVolumeClaim\":{\".\":{},\"f:claimName\":{}}}}}}",
                            },
                            Subresource: "",
                        },
                        {
     ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

Jan 28 13:17:48.752: FAIL: failed to start pod: timed out while waiting for pod csi-mock-volumes-8888/pvc-volume-tester-z4mbj to be running

Full Stack Trace
k8s.io/kubernetes/test/e2e/storage.glob..func2.14.1()
	test/e2e/storage/csi_mock_volume.go:1211 +0x626
�[1mSTEP:�[0m Deleting pod pvc-volume-tester-z4mbj �[38;5;243m01/28/23 13:17:48.752�[0m
Jan 28 13:17:48.752: INFO: Deleting pod "pvc-volume-tester-z4mbj" in namespace "csi-mock-volumes-8888"
Jan 28 13:17:48.968: INFO: Wait up to 5m0s for pod "pvc-volume-tester-z4mbj" to be fully deleted
Jan 28 13:18:00.268: INFO: >>> kubeConfig: /root/.kube/config
Jan 28 13:18:00.269: INFO: ExecWithOptions: Clientset creation
Jan 28 13:18:00.269: INFO: ExecWithOptions: execute(POST https://api.e2e-e2e-kops-grid-cilium-etcd-u2204-k25-ko26.test-cncf-aws.k8s.io/api/v1/namespaces/csi-mock-volumes-8888-4534/pods/csi-mockplugin-0/exec?command=rm&command=-rf&command=%2Fvar%2Flib%2Fkubelet%2Fpods%2F0818f735-5dda-41ba-8a33-f952e95834a3%2Fvolumes%2Fkubernetes.io~csi%2Fpvc-79c1692e-f2a8-4e5a-94d7-e580686588ac%2Fmount&container=busybox&container=busybox&stderr=true&stdout=true)
I0128 13:18:01.759312    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeUnpublishVolume","Request":{"volume_id":"4","target_path":"/var/lib/kubelet/pods/0818f735-5dda-41ba-8a33-f952e95834a3/volumes/kubernetes.io~csi/pvc-79c1692e-f2a8-4e5a-94d7-e580686588ac/mount"},"Response":{},"Error":"","FullError":null}
I0128 13:18:02.001330    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeGetCapabilities","Request":{},"Response":{"capabilities":[{"Type":{"Rpc":{}}},{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":4}}}]},"Error":"","FullError":null}
I0128 13:18:02.256421    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Node/NodeUnstageVolume","Request":{"volume_id":"4","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/csi-mock-csi-mock-volumes-8888/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a/globalmount"},"Response":{},"Error":"","FullError":null}
�[1mSTEP:�[0m Deleting claim pvc-vlt5c �[38;5;243m01/28/23 13:18:03.398�[0m
Jan 28 13:18:03.829: INFO: Waiting up to 2m0s for PersistentVolume pvc-79c1692e-f2a8-4e5a-94d7-e580686588ac to get deleted
I0128 13:18:03.851279    6708 csi.go:435] gRPCCall: {"Method":"/csi.v1.Controller/DeleteVolume","Request":{"volume_id":"4"},"Response":{},"Error":"","FullError":null}
Jan 28 13:18:04.044: INFO: PersistentVolume pvc-79c1692e-f2a8-4e5a-94d7-e580686588ac found and phase=Released (214.346684ms)
Jan 28 13:18:06.259: INFO: PersistentVolume pvc-79c1692e-f2a8-4e5a-94d7-e580686588ac was removed
�[1mSTEP:�[0m Deleting storageclass csi-mock-volumes-8888-sctwtdb �[38;5;243m01/28/23 13:18:06.259�[0m
�[1mSTEP:�[0m Cleaning up resources �[38;5;243m01/28/23 13:18:06.475�[0m
�[1mSTEP:�[0m deleting the test namespace: csi-mock-volumes-8888 �[38;5;243m01/28/23 13:18:15.127�[0m
�[1mSTEP:�[0m Waiting for namespaces [csi-mock-volumes-8888] to vanish �[38;5;243m01/28/23 13:18:15.343�[0m
�[1mSTEP:�[0m uninstalling csi mock driver �[38;5;243m01/28/23 13:18:21.56�[0m
Jan 28 13:18:21.560: INFO: deleting *v1.ServiceAccount: csi-mock-volumes-8888-4534/csi-attacher
Jan 28 13:18:21.776: INFO: deleting *v1.ClusterRole: external-attacher-runner-csi-mock-volumes-8888
Jan 28 13:18:21.994: INFO: deleting *v1.ClusterRoleBinding: csi-attacher-role-csi-mock-volumes-8888
Jan 28 13:18:22.211: INFO: deleting *v1.Role: csi-mock-volumes-8888-4534/external-attacher-cfg-csi-mock-volumes-8888
Jan 28 13:18:22.427: INFO: deleting *v1.RoleBinding: csi-mock-volumes-8888-4534/csi-attacher-role-cfg
Jan 28 13:18:22.643: INFO: deleting *v1.ServiceAccount: csi-mock-volumes-8888-4534/csi-provisioner
Jan 28 13:18:22.874: INFO: deleting *v1.ClusterRole: external-provisioner-runner-csi-mock-volumes-8888
Jan 28 13:18:23.147: INFO: deleting *v1.ClusterRoleBinding: csi-provisioner-role-csi-mock-volumes-8888
Jan 28 13:18:23.363: INFO: deleting *v1.Role: csi-mock-volumes-8888-4534/external-provisioner-cfg-csi-mock-volumes-8888
Jan 28 13:18:23.580: INFO: deleting *v1.RoleBinding: csi-mock-volumes-8888-4534/csi-provisioner-role-cfg
Jan 28 13:18:23.797: INFO: deleting *v1.ServiceAccount: csi-mock-volumes-8888-4534/csi-resizer
Jan 28 13:18:24.014: INFO: deleting *v1.ClusterRole: external-resizer-runner-csi-mock-volumes-8888
Jan 28 13:18:24.231: INFO: deleting *v1.ClusterRoleBinding: csi-resizer-role-csi-mock-volumes-8888
Jan 28 13:18:24.448: INFO: deleting *v1.Role: csi-mock-volumes-8888-4534/external-resizer-cfg-csi-mock-volumes-8888
Jan 28 13:18:24.667: INFO: deleting *v1.RoleBinding: csi-mock-volumes-8888-4534/csi-resizer-role-cfg
Jan 28 13:18:24.884: INFO: deleting *v1.ServiceAccount: csi-mock-volumes-8888-4534/csi-snapshotter
Jan 28 13:18:25.100: INFO: deleting *v1.ClusterRole: external-snapshotter-runner-csi-mock-volumes-8888
Jan 28 13:18:25.317: INFO: deleting *v1.ClusterRoleBinding: csi-snapshotter-role-csi-mock-volumes-8888
Jan 28 13:18:25.534: INFO: deleting *v1.Role: csi-mock-volumes-8888-4534/external-snapshotter-leaderelection-csi-mock-volumes-8888
Jan 28 13:18:25.751: INFO: deleting *v1.RoleBinding: csi-mock-volumes-8888-4534/external-snapshotter-leaderelection
Jan 28 13:18:25.968: INFO: deleting *v1.ServiceAccount: csi-mock-volumes-8888-4534/csi-mock
Jan 28 13:18:26.185: INFO: deleting *v1.ClusterRoleBinding: csi-controller-attacher-role-csi-mock-volumes-8888
Jan 28 13:18:26.402: INFO: deleting *v1.ClusterRoleBinding: csi-controller-provisioner-role-csi-mock-volumes-8888
Jan 28 13:18:26.618: INFO: deleting *v1.ClusterRoleBinding: csi-controller-cluster-driver-registrar-role-csi-mock-volumes-8888
Jan 28 13:18:26.835: INFO: deleting *v1.ClusterRoleBinding: psp-csi-controller-driver-registrar-role-csi-mock-volumes-8888
Jan 28 13:18:27.052: INFO: deleting *v1.ClusterRoleBinding: csi-controller-resizer-role-csi-mock-volumes-8888
Jan 28 13:18:27.269: INFO: deleting *v1.ClusterRoleBinding: csi-controller-snapshotter-role-csi-mock-volumes-8888
Jan 28 13:18:27.501: INFO: deleting *v1.StorageClass: csi-mock-sc-csi-mock-volumes-8888
Jan 28 13:18:27.725: INFO: deleting *v1.StatefulSet: csi-mock-volumes-8888-4534/csi-mockplugin
Jan 28 13:18:27.943: INFO: deleting *v1.CSIDriver: csi-mock-csi-mock-volumes-8888
�[1mSTEP:�[0m deleting the driver namespace: csi-mock-volumes-8888-4534 �[38;5;243m01/28/23 13:18:28.172�[0m
�[1mSTEP:�[0m Waiting for namespaces [csi-mock-volumes-8888-4534] to vanish �[38;5;243m01/28/23 13:18:28.389�[0m
[AfterEach] [sig-storage] CSI mock volume
  test/e2e/framework/framework.go:187
Jan 28 13:19:12.825: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

Kubernetes e2e suite [It] [sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should support non-existent path 5m13s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-storage\]\sIn\-tree\sVolumes\s\[Driver\:\shostPathSymlink\]\s\[Testpattern\:\sInline\-volume\s\(default\sfs\)\]\ssubPath\sshould\ssupport\snon\-existent\spath$'

test/e2e/storage/drivers/in_tree.go:955
k8s.io/kubernetes/test/e2e/storage/drivers.(*hostPathSymlinkDriver).CreateVolume(0xc0006b4180?, 0xc00358ce40, {0xc001142ea0?, 0xc000234a08?})
	test/e2e/storage/drivers/in_tree.go:955 +0x9e5
k8s.io/kubernetes/test/e2e/storage/framework.CreateVolume({0x7c51fe8, 0xc001142ea0}, 0xc00122c000?, {0x7377570, 0xc})
	test/e2e/storage/framework/driver_operations.go:43 +0xd2
k8s.io/kubernetes/test/e2e/storage/framework.CreateVolumeResource({0x7c51fe8, 0xc001142ea0}, 0xc00358ce40, {{0x73cac5f, 0x1a}, {0x0, 0x0}, {0x7377570, 0xc}, {0x0, ...}, ...}, ...)
	test/e2e/storage/framework/volume_resource.go:65 +0x225
k8s.io/kubernetes/test/e2e/storage/testsuites.(*subPathTestSuite).DefineTests.func1()
	test/e2e/storage/testsuites/subpath.go:128 +0x28e
k8s.io/kubernetes/test/e2e/storage/testsuites.(*subPathTestSuite).DefineTests.func3()
	test/e2e/storage/testsuites/subpath.go:196 +0x4d
				{"msg":"FAILED [sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should support non-existent path","completed":35,"skipped":209,"failed":1,"failures":["[sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should support non-existent path"]}

[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/framework/framework.go:186
�[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m01/28/23 13:13:03.602�[0m
Jan 28 13:13:03.603: INFO: >>> kubeConfig: /root/.kube/config
�[1mSTEP:�[0m Building a namespace api object, basename provisioning �[38;5;243m01/28/23 13:13:03.603�[0m
�[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m01/28/23 13:13:04.251�[0m
�[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m01/28/23 13:13:04.681�[0m
[It] should support non-existent path
  test/e2e/storage/testsuites/subpath.go:195
Jan 28 13:13:05.111: INFO: In-tree plugin kubernetes.io/host-path is not migrated, not validating any metrics
Jan 28 13:13:05.551: INFO: Waiting up to 5m0s for pod "hostpath-symlink-prep-provisioning-2881" in namespace "provisioning-2881" to be "Succeeded or Failed"
Jan 28 13:13:05.769: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 217.999124ms
Jan 28 13:13:07.987: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2.436319004s
Jan 28 13:13:09.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4.433706316s
Jan 28 13:13:11.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 6.433996019s
Jan 28 13:13:13.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 8.434503371s
Jan 28 13:13:15.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 10.434437411s
Jan 28 13:13:17.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 12.434667274s
Jan 28 13:13:19.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 14.434097548s
Jan 28 13:13:21.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 16.43428099s
Jan 28 13:13:23.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 18.434146083s
Jan 28 13:13:25.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 20.433851291s
Jan 28 13:13:27.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 22.4347682s
Jan 28 13:13:29.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 24.433899971s
Jan 28 13:13:31.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 26.434285232s
Jan 28 13:13:33.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 28.434454621s
Jan 28 13:13:35.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 30.433779584s
Jan 28 13:13:37.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 32.433708713s
Jan 28 13:13:39.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 34.43374846s
Jan 28 13:13:41.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 36.434513088s
Jan 28 13:13:43.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 38.434739362s
Jan 28 13:13:45.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 40.433596248s
Jan 28 13:13:47.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 42.433916796s
Jan 28 13:13:49.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 44.433939043s
Jan 28 13:13:51.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 46.434219887s
Jan 28 13:13:53.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 48.435509047s
Jan 28 13:13:55.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 50.433944251s
Jan 28 13:13:57.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 52.433706463s
Jan 28 13:13:59.987: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 54.436274449s
Jan 28 13:14:01.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 56.434988252s
Jan 28 13:14:03.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 58.434422709s
Jan 28 13:14:05.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m0.433883551s
Jan 28 13:14:07.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m2.434992839s
Jan 28 13:14:09.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.434497633s
Jan 28 13:14:11.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.43416251s
Jan 28 13:14:13.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m8.433879577s
Jan 28 13:14:15.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m10.434088193s
Jan 28 13:14:17.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m12.434054076s
Jan 28 13:14:19.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.434058593s
Jan 28 13:14:21.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.4345691s
Jan 28 13:14:23.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.434010487s
Jan 28 13:14:25.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.434127008s
Jan 28 13:14:27.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.434623295s
Jan 28 13:14:29.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.433571277s
Jan 28 13:14:31.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.433909326s
Jan 28 13:14:33.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.434834906s
Jan 28 13:14:35.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.434173721s
Jan 28 13:14:37.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.435157025s
Jan 28 13:14:39.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m34.434384055s
Jan 28 13:14:42.000: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m36.449365543s
Jan 28 13:14:43.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m38.434915779s
Jan 28 13:14:45.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m40.4339093s
Jan 28 13:14:47.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m42.434265383s
Jan 28 13:14:49.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m44.43402129s
Jan 28 13:14:51.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m46.434142447s
Jan 28 13:14:53.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m48.434275166s
Jan 28 13:14:55.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m50.433842352s
Jan 28 13:14:57.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m52.434369853s
Jan 28 13:14:59.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m54.434524106s
Jan 28 13:15:01.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m56.433966715s
Jan 28 13:15:03.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 1m58.434156372s
Jan 28 13:15:05.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m0.434348912s
Jan 28 13:15:07.989: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m2.43843983s
Jan 28 13:15:09.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m4.434457433s
Jan 28 13:15:11.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m6.433834545s
Jan 28 13:15:13.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m8.434177917s
Jan 28 13:15:15.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m10.435203537s
Jan 28 13:15:17.988: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m12.436971199s
Jan 28 13:15:19.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m14.43429638s
Jan 28 13:15:21.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m16.433724601s
Jan 28 13:15:23.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m18.433625492s
Jan 28 13:15:25.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m20.434398016s
Jan 28 13:15:27.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m22.433734119s
Jan 28 13:15:29.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m24.433724636s
Jan 28 13:15:31.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m26.434196129s
Jan 28 13:15:33.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m28.434135586s
Jan 28 13:15:35.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m30.434273359s
Jan 28 13:15:37.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m32.435569131s
Jan 28 13:15:39.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m34.433956421s
Jan 28 13:15:41.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m36.434444092s
Jan 28 13:15:43.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m38.434526216s
Jan 28 13:15:45.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m40.433825198s
Jan 28 13:15:47.992: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m42.441260253s
Jan 28 13:15:49.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m44.434311338s
Jan 28 13:15:51.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m46.433820428s
Jan 28 13:15:53.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m48.433618474s
Jan 28 13:15:55.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m50.433587944s
Jan 28 13:15:57.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m52.43571979s
Jan 28 13:15:59.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m54.43365902s
Jan 28 13:16:01.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m56.433872198s
Jan 28 13:16:03.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 2m58.434893643s
Jan 28 13:16:05.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m0.433828678s
Jan 28 13:16:07.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m2.434352645s
Jan 28 13:16:09.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m4.434330333s
Jan 28 13:16:11.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m6.434537134s
Jan 28 13:16:13.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m8.434602176s
Jan 28 13:16:15.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m10.434955832s
Jan 28 13:16:17.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m12.433682133s
Jan 28 13:16:19.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m14.434001828s
Jan 28 13:16:21.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m16.434183335s
Jan 28 13:16:23.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m18.433384181s
Jan 28 13:16:25.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m20.433580535s
Jan 28 13:16:27.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m22.434459564s
Jan 28 13:16:29.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m24.435238592s
Jan 28 13:16:31.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m26.433829649s
Jan 28 13:16:33.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m28.434278983s
Jan 28 13:16:35.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m30.433448956s
Jan 28 13:16:37.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m32.434062762s
Jan 28 13:16:39.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m34.433671252s
Jan 28 13:16:41.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m36.433461085s
Jan 28 13:16:43.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m38.43416538s
Jan 28 13:16:45.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m40.43400027s
Jan 28 13:16:47.987: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m42.43619734s
Jan 28 13:16:49.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m44.433879871s
Jan 28 13:16:51.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m46.433380899s
Jan 28 13:16:53.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m48.433169241s
Jan 28 13:16:55.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m50.433377419s
Jan 28 13:16:57.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m52.43367896s
Jan 28 13:16:59.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m54.433681273s
Jan 28 13:17:01.987: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m56.435814991s
Jan 28 13:17:03.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 3m58.434308427s
Jan 28 13:17:05.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m0.433746405s
Jan 28 13:17:07.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m2.433227787s
Jan 28 13:17:09.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m4.434201857s
Jan 28 13:17:11.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m6.433736562s
Jan 28 13:17:13.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m8.433778174s
Jan 28 13:17:15.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m10.433258885s
Jan 28 13:17:17.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m12.433930626s
Jan 28 13:17:19.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m14.43356422s
Jan 28 13:17:21.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m16.434581017s
Jan 28 13:17:23.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m18.434329427s
Jan 28 13:17:25.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m20.43331424s
Jan 28 13:17:28.006: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m22.454774164s
Jan 28 13:17:29.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m24.433591467s
Jan 28 13:17:31.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m26.433574504s
Jan 28 13:17:33.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.434092609s
Jan 28 13:17:35.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.433630959s
Jan 28 13:17:37.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.435661436s
Jan 28 13:17:39.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.433587617s
Jan 28 13:17:41.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.43355698s
Jan 28 13:17:43.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m38.434426057s
Jan 28 13:17:45.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m40.434356504s
Jan 28 13:17:47.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m42.434521833s
Jan 28 13:17:49.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m44.433706574s
Jan 28 13:17:51.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m46.434581171s
Jan 28 13:17:53.986: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.435016908s
Jan 28 13:17:55.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.434467192s
Jan 28 13:17:57.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.434651723s
Jan 28 13:17:59.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.433851838s
Jan 28 13:18:01.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.434266113s
Jan 28 13:18:03.985: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.434216692s
Jan 28 13:18:05.984: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.433293131s
Jan 28 13:18:06.199: INFO: Pod "hostpath-symlink-prep-provisioning-2881": Phase="Pending", Reason="", readiness=false. Elapsed: 5m0.648465388s
Jan 28 13:18:06.200: INFO: Unexpected error: while waiting for hostPath init pod to succeed: 
    <*pod.timeoutError | 0xc003d7dbc0>: {
        msg: "timed out while waiting for pod provisioning-2881/hostpath-symlink-prep-provisioning-2881 to be Succeeded or Failed",
        observedObjects: [
            <*v1.Pod | 0xc001868400>{
                TypeMeta: {Kind: "", APIVersion: ""},
                ObjectMeta: {
                    Name: "hostpath-symlink-prep-provisioning-2881",
                    GenerateName: "",
                    Namespace: "provisioning-2881",
                    SelfLink: "",
                    UID: "0ef2f13f-2c38-4323-a58f-a7341682b41f",
                    ResourceVersion: "39024",
                    Generation: 0,
                    CreationTimestamp: {
                        Time: {
                            wall: 0,
                            ext: 63810508385,
                            loc: {
                                name: "Local",
                                zone: [
                                    {name: "UTC", offset: 0, isDST: false},
                                ],
                                tx: [
                                    {
                                        when: -9223372036854775808,
                                        index: 0,
                                        isstd: false,
                                        isutc: false,
                                    },
                                ],
                                extend: "UTC0",
                                cacheStart: 9223372036854775807,
                                cacheEnd: 9223372036854775807,
                                cacheZone: {name: "UTC", offset: 0, isDST: false},
                            },
                        },
                    },
                    DeletionTimestamp: nil,
                    DeletionGracePeriodSeconds: nil,
                    Labels: nil,
                    Annotations: nil,
                    OwnerReferences: nil,
                    Finalizers: nil,
                    ManagedFields: [
                        {
                            Manager: "e2e.test",
                            Operation: "Update",
                            APIVersion: "v1",
                            Time: {
                                Time: {
                                    wall: 0,
                                    ext: 63810508385,
                                    loc: {
                                        name: "Local",
                                        zone: [...],
                                        tx: [...],
                                        extend: "UTC0",
                                        cacheStart: 9223372036854775807,
                                        cacheEnd: 9223372036854775807,
                                        cacheZone: {name: ..., offset: ..., isDST: ...},
                                    },
                                },
                            },
                            FieldsType: "FieldsV1",
                            FieldsV1: {
                                Raw: "{\"f:spec\":{\"f:containers\":{\"k:{\\\"name\\\":\\\"init-volume-provisioning-2881\\\"}\":{\".\":{},\"f:command\":{},\"f:image\":{},\"f:imagePullPolicy\":{},\"f:name\":{},\"f:resources\":{},\"f:securityContext\":{\".\":{},\"f:privileged\":{}},\"f:terminationMessagePath\":{},\"f:terminationMessagePolicy\":{},\"f:volumeMounts\":{\".\":{},\"k:{\\\"mountPath\\\":\\\"/tmp\\\"}\":{\".\":{},\"f:mountPath\":{},\"f:name\":{}}}}},\"f:dnsPolicy\":{},\"f:enableServiceLinks\":{},\"f:nodeName\":{},\"f:restartPolicy\":{},\"f:schedulerName\":{},\"f:securityContext\":{},\"f:terminationGracePeriodSeconds\":{},\"f:volumes\":{\".\":{},\"k:{\\\"name\\\":\\\"test-volume\\\"}\":{\".\":{},\"f:hostPath\":{\".\":{},\"f:path\":{},\"f:type\":{}},\"f:name\":{}}}}}",
                            },
                            Subresource: "",
                        },
                        {
                            Manager: "kub...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

Jan 28 13:18:06.200: FAIL: while waiting for hostPath init pod to succeed: timed out while waiting for pod provisioning-2881/hostpath-symlink-prep-provisioning-2881 to be Succeeded or Failed

Full Stack Trace
k8s.io/kubernetes/test/e2e/storage/drivers.(*hostPathSymlinkDriver).CreateVolume(0xc0006b4180?, 0xc00358ce40, {0xc001142ea0?, 0xc000234a08?})
	test/e2e/storage/drivers/in_tree.go:955 +0x9e5
k8s.io/kubernetes/test/e2e/storage/framework.CreateVolume({0x7c51fe8, 0xc001142ea0}, 0xc00122c000?, {0x7377570, 0xc})
	test/e2e/storage/framework/driver_operations.go:43 +0xd2
k8s.io/kubernetes/test/e2e/storage/framework.CreateVolumeResource({0x7c51fe8, 0xc001142ea0}, 0xc00358ce40, {{0x73cac5f, 0x1a}, {0x0, 0x0}, {0x7377570, 0xc}, {0x0, ...}, ...}, ...)
	test/e2e/storage/framework/volume_resource.go:65 +0x225
k8s.io/kubernetes/test/e2e/storage/testsuites.(*subPathTestSuite).DefineTests.func1()
	test/e2e/storage/testsuites/subpath.go:128 +0x28e
k8s.io/kubernetes/test/e2e/storage/testsuites.(*subPathTestSuite).DefineTests.func3()
	test/e2e/storage/testsuites/subpath.go:196 +0x4d
[AfterEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/framework/framework.go:187
�[1mSTEP:�[0m Collecting events from namespace "provisioning-2881". �[38;5;243m01/28/23 13:18:06.201�[0m
�[1mSTEP:�[0m Found 2 events. �[38;5;243m01/28/23 13:18:06.416�[0m
Jan 28 13:18:06.416: INFO: At 2023-01-28 13:14:05 +0000 UTC - event for hostpath-symlink-prep-provisioning-2881: {kubelet i-011ed997f238bea92} FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9725fba67d3b9125954f887ae1c821686f27d1f6660f93f39bd9bc839f2cb46e": plugin type="cilium-cni" name="cilium" failed (add): unable to connect to Cilium daemon: failed to create cilium agent client after 30.000000 seconds timeout: Get "http:///var/run/cilium/cilium.sock/v1/config": dial unix /var/run/cilium/cilium.sock: connect: connection refused
Is the agent running?
Jan 28 13:18:06.416: INFO: At 2023-01-28 13:14:06 +0000 UTC - event for hostpath-symlink-prep-provisioning-2881: {kubelet i-011ed997f238bea92} SandboxChanged: Pod sandbox changed, it will be killed and re-created.
Jan 28 13:18:06.631: INFO: POD                                      NODE                 PHASE    GRACE  CONDITIONS
Jan 28 13:18:06.632: INFO: hostpath-symlink-prep-provisioning-2881  i-011ed997f238bea92  Pending         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:13:05 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:13:05 +0000 UTC ContainersNotReady containers with unready status: [init-volume-provisioning-2881]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:13:05 +0000 UTC ContainersNotReady containers with unready status: [init-volume-provisioning-2881]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-01-28 13:13:05 +0000 UTC  }]
Jan 28 13:18:06.632: INFO: 
Jan 28 13:18:06.848: INFO: Unable to fetch provisioning-2881/hostpath-symlink-prep-provisioning-2881/init-volume-provisioning-2881 logs: the server rejected our request for an unknown reason (get pods hostpath-symlink-prep-provisioning-2881)
Jan 28 13:18:07.065: INFO: 
Logging node info for node i-00e83413dfa35ee6b
Jan 28 13:18:07.283: INFO: Node Info: &Node{ObjectMeta:{i-00e83413dfa35ee6b    1a086b23-20ac-414b-98eb-191b96574ca7 45888 0 2023-01-28 12:52:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-00e83413dfa35ee6b kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-00e83413dfa35ee6b topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9664":"i-00e83413dfa35ee6b","ebs.csi.aws.com":"i-00e83413dfa35ee6b"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 12:52:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 12:52:27 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 12:52:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 12:52:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 12:52:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-01-28 13:17:12 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 13:18:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-00e83413dfa35ee6b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:18 +0000 UTC,LastTransitionTime:2023-01-28 12:52:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:18 +0000 UTC,LastTransitionTime:2023-01-28 12:52:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:18 +0000 UTC,LastTransitionTime:2023-01-28 12:52:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 13:17:18 +0000 UTC,LastTransitionTime:2023-01-28 12:52:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.66,},NodeAddress{Type:ExternalIP,Address:54.169.3.124,},NodeAddress{Type:InternalDNS,Address:i-00e83413dfa35ee6b.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-00e83413dfa35ee6b.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-169-3-124.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dd38b50ae71201c15829db9e8dbeb,SystemUUID:ec2dd38b-50ae-7120-1c15-829db9e8dbeb,BootID:603bf541-7f58-4980-9ee5-4bcd00e1d1ef,KernelVersion:5.15.0-1028-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.25.6,KubeProxyVersion:v1.25.6,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.6],SizeBytes:63291593,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c registry.k8s.io/e2e-test-images/nautilus:1.5],SizeBytes:49642095,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:b9e2958a3dd879e3cf11142228c6d073d0fc4ea2e857c3be6f4fb0ab5fb2c937 registry.k8s.io/e2e-test-images/nonroot:1.2],SizeBytes:17748301,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-088ae15ae985c4286],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-088ae15ae985c4286,DevicePath:,},},Config:nil,},}
Jan 28 13:18:07.283: INFO: 
Logging kubelet events for node i-00e83413dfa35ee6b
Jan 28 13:18:07.501: INFO: 
Logging pods the kubelet thinks is on node i-00e83413dfa35ee6b
Jan 28 13:18:07.724: INFO: netserver-0 started at 2023-01-28 13:12:48 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:07.724: INFO: csi-mockplugin-0 started at 2023-01-28 13:16:53 +0000 UTC (0+4 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Container busybox ready: true, restart count 0
Jan 28 13:18:07.724: INFO: 	Container csi-provisioner ready: true, restart count 2
Jan 28 13:18:07.724: INFO: 	Container driver-registrar ready: true, restart count 0
Jan 28 13:18:07.724: INFO: 	Container mock ready: true, restart count 0
Jan 28 13:18:07.724: INFO: netserver-0 started at 2023-01-28 13:14:01 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:07.724: INFO: pod-subpath-test-preprovisionedpv-wmm7 started at 2023-01-28 13:17:46 +0000 UTC (1+1 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Init container init-volume-preprovisionedpv-wmm7 ready: true, restart count 0
Jan 28 13:18:07.724: INFO: 	Container test-container-subpath-preprovisionedpv-wmm7 ready: true, restart count 0
Jan 28 13:18:07.724: INFO: ss2-2 started at 2023-01-28 13:13:22 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:07.724: INFO: pod-client started at 2023-01-28 13:13:31 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Container pod-client ready: true, restart count 0
Jan 28 13:18:07.724: INFO: hostexec-i-00e83413dfa35ee6b-59mg8 started at 2023-01-28 13:17:30 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 28 13:18:07.724: INFO: hostexec-i-00e83413dfa35ee6b-hbqnp started at 2023-01-28 13:17:47 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 28 13:18:07.724: INFO: ebs-csi-node-qfplb started at 2023-01-28 12:52:28 +0000 UTC (0+3 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 28 13:18:07.724: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 28 13:18:07.724: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 28 13:18:07.724: INFO: ss-1 started at 2023-01-28 13:17:10 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:07.724: INFO: webserver-54977698-5pcd4 started at 2023-01-28 13:12:49 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Container httpd ready: true, restart count 0
Jan 28 13:18:07.724: INFO: webserver-54977698-ql29p started at 2023-01-28 13:13:08 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Container httpd ready: true, restart count 0
Jan 28 13:18:07.724: INFO: service-proxy-disabled-dt7t6 started at 2023-01-28 13:13:59 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Container service-proxy-disabled ready: true, restart count 0
Jan 28 13:18:07.724: INFO: cilium-p9r7t started at 2023-01-28 12:52:28 +0000 UTC (1+1 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 28 13:18:07.724: INFO: 	Container cilium-agent ready: true, restart count 3
Jan 28 13:18:07.724: INFO: pod-subpath-test-preprovisionedpv-htrf started at 2023-01-28 13:18:01 +0000 UTC (2+1 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Init container init-volume-preprovisionedpv-htrf ready: true, restart count 0
Jan 28 13:18:07.724: INFO: 	Init container test-init-subpath-preprovisionedpv-htrf ready: true, restart count 0
Jan 28 13:18:07.724: INFO: 	Container test-container-subpath-preprovisionedpv-htrf ready: false, restart count 0
Jan 28 13:18:07.724: INFO: csi-mockplugin-0 started at 2023-01-28 13:17:58 +0000 UTC (0+3 container statuses recorded)
Jan 28 13:18:07.724: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 28 13:18:07.724: INFO: 	Container driver-registrar ready: true, restart count 0
Jan 28 13:18:07.724: INFO: 	Container mock ready: true, restart count 0
Jan 28 13:18:08.499: INFO: 
Latency metrics for node i-00e83413dfa35ee6b
Jan 28 13:18:08.499: INFO: 
Logging node info for node i-011ed997f238bea92
Jan 28 13:18:08.714: INFO: Node Info: &Node{ObjectMeta:{i-011ed997f238bea92    b3924a8c-3c07-4e89-9655-6dede0550248 43562 0 2023-01-28 12:52:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-011ed997f238bea92 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-011ed997f238bea92 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-5559":"i-011ed997f238bea92","csi-mock-csi-mock-volumes-8888":"csi-mock-csi-mock-volumes-8888","ebs.csi.aws.com":"i-011ed997f238bea92"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 12:52:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 12:52:35 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 12:52:35 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 12:52:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 12:52:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-01-28 13:15:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-01-28 13:16:05 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-011ed997f238bea92,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 13:15:57 +0000 UTC,LastTransitionTime:2023-01-28 12:52:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 13:15:57 +0000 UTC,LastTransitionTime:2023-01-28 12:52:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 13:15:57 +0000 UTC,LastTransitionTime:2023-01-28 12:52:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 13:15:57 +0000 UTC,LastTransitionTime:2023-01-28 12:52:55 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.54,},NodeAddress{Type:ExternalIP,Address:54.254.241.213,},NodeAddress{Type:InternalDNS,Address:i-011ed997f238bea92.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-011ed997f238bea92.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-241-213.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2847264721fc60b1e2b62ff5f906af,SystemUUID:ec284726-4721-fc60-b1e2-b62ff5f906af,BootID:c04be1bb-4aec-4103-8d4e-d3b345c3abb6,KernelVersion:5.15.0-1028-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.25.6,KubeProxyVersion:v1.25.6,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.6],SizeBytes:63291593,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-5559^3feeda26-9f0d-11ed-84c3-66b7a3d0f20d kubernetes.io/csi/ebs.csi.aws.com^vol-00e9c4b4c411a4354],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-00e9c4b4c411a4354,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5559^3feeda26-9f0d-11ed-84c3-66b7a3d0f20d,DevicePath:,},},Config:nil,},}
Jan 28 13:18:08.714: INFO: 
Logging kubelet events for node i-011ed997f238bea92
Jan 28 13:18:08.932: INFO: 
Logging pods the kubelet thinks is on node i-011ed997f238bea92
Jan 28 13:18:09.165: INFO: coredns-85d58b74c8-9rvwc started at 2023-01-28 12:53:09 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container coredns ready: true, restart count 0
Jan 28 13:18:09.165: INFO: hostexec-i-011ed997f238bea92-hxw5m started at 2023-01-28 13:13:20 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 28 13:18:09.165: INFO: netserver-1 started at 2023-01-28 13:14:01 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container webserver ready: false, restart count 0
Jan 28 13:18:09.165: INFO: hostexec-i-011ed997f238bea92-fb4sv started at 2023-01-28 13:14:25 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 28 13:18:09.165: INFO: ebs-csi-node-8wh45 started at 2023-01-28 12:52:36 +0000 UTC (0+3 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 28 13:18:09.165: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 28 13:18:09.165: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 28 13:18:09.165: INFO: pod-server-1 started at 2023-01-28 13:13:36 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container agnhost-container ready: false, restart count 0
Jan 28 13:18:09.165: INFO: pod-subpath-test-preprovisionedpv-hkbx started at 2023-01-28 13:14:33 +0000 UTC (2+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Init container init-volume-preprovisionedpv-hkbx ready: true, restart count 0
Jan 28 13:18:09.165: INFO: 	Init container test-init-volume-preprovisionedpv-hkbx ready: true, restart count 0
Jan 28 13:18:09.165: INFO: 	Container test-container-subpath-preprovisionedpv-hkbx ready: false, restart count 0
Jan 28 13:18:09.165: INFO: csi-hostpathplugin-0 started at 2023-01-28 13:11:15 +0000 UTC (0+7 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 28 13:18:09.165: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 28 13:18:09.165: INFO: 	Container csi-resizer ready: true, restart count 0
Jan 28 13:18:09.165: INFO: 	Container csi-snapshotter ready: true, restart count 0
Jan 28 13:18:09.165: INFO: 	Container hostpath ready: true, restart count 0
Jan 28 13:18:09.165: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 28 13:18:09.165: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 28 13:18:09.165: INFO: netserver-1 started at 2023-01-28 13:12:48 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container webserver ready: false, restart count 0
Jan 28 13:18:09.165: INFO: webserver-54977698-4sqcm started at 2023-01-28 13:13:07 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container httpd ready: true, restart count 0
Jan 28 13:18:09.165: INFO: pod-c161f54c-125e-4395-be49-581387fc94ee started at 2023-01-28 13:13:26 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container write-pod ready: false, restart count 0
Jan 28 13:18:09.165: INFO: exec-volume-test-preprovisionedpv-qf68 started at 2023-01-28 13:13:46 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container exec-container-preprovisionedpv-qf68 ready: false, restart count 0
Jan 28 13:18:09.165: INFO: csi-hostpathplugin-0 started at 2023-01-28 13:14:05 +0000 UTC (0+7 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container csi-attacher ready: false, restart count 0
Jan 28 13:18:09.165: INFO: 	Container csi-provisioner ready: false, restart count 0
Jan 28 13:18:09.165: INFO: 	Container csi-resizer ready: false, restart count 0
Jan 28 13:18:09.165: INFO: 	Container csi-snapshotter ready: false, restart count 0
Jan 28 13:18:09.165: INFO: 	Container hostpath ready: false, restart count 0
Jan 28 13:18:09.165: INFO: 	Container liveness-probe ready: false, restart count 0
Jan 28 13:18:09.165: INFO: 	Container node-driver-registrar ready: false, restart count 0
Jan 28 13:18:09.165: INFO: pod-subpath-test-preprovisionedpv-mnph started at 2023-01-28 13:16:47 +0000 UTC (1+2 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Init container test-init-subpath-preprovisionedpv-mnph ready: false, restart count 0
Jan 28 13:18:09.165: INFO: 	Container test-container-subpath-preprovisionedpv-mnph ready: false, restart count 0
Jan 28 13:18:09.165: INFO: 	Container test-container-volume-preprovisionedpv-mnph ready: false, restart count 0
Jan 28 13:18:09.165: INFO: hostpath-symlink-prep-provisioning-2881 started at 2023-01-28 13:13:05 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container init-volume-provisioning-2881 ready: false, restart count 0
Jan 28 13:18:09.165: INFO: service-proxy-disabled-rvj8q started at 2023-01-28 13:13:59 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container service-proxy-disabled ready: true, restart count 0
Jan 28 13:18:09.165: INFO: ss-1 started at 2023-01-28 13:15:53 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container webserver ready: false, restart count 0
Jan 28 13:18:09.165: INFO: cilium-4dcmk started at 2023-01-28 12:52:36 +0000 UTC (1+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 28 13:18:09.165: INFO: 	Container cilium-agent ready: true, restart count 6
Jan 28 13:18:09.165: INFO: hostexec-i-011ed997f238bea92-st2h8 started at 2023-01-28 13:13:30 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 28 13:18:09.165: INFO: hostexec-i-011ed997f238bea92-6ktg7 started at 2023-01-28 13:16:34 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 28 13:18:09.165: INFO: csi-mockplugin-0 started at 2023-01-28 13:12:29 +0000 UTC (0+4 container statuses recorded)
Jan 28 13:18:09.165: INFO: 	Container busybox ready: true, restart count 0
Jan 28 13:18:09.165: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 28 13:18:09.165: INFO: 	Container driver-registrar ready: true, restart count 0
Jan 28 13:18:09.165: INFO: 	Container mock ready: true, restart count 0
Jan 28 13:18:12.225: INFO: 
Latency metrics for node i-011ed997f238bea92
Jan 28 13:18:12.225: INFO: 
Logging node info for node i-04736cc52ac74b9d6
Jan 28 13:18:12.441: INFO: Node Info: &Node{ObjectMeta:{i-04736cc52ac74b9d6    f886f5d3-f71b-465b-8bf8-d2ad6142224c 45452 0 2023-01-28 12:52:30 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-04736cc52ac74b9d6 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-04736cc52ac74b9d6 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-04736cc52ac74b9d6"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 12:52:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 12:52:30 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 12:52:30 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-01-28 12:52:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-01-28 12:53:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 13:17:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-01-28 13:17:32 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-04736cc52ac74b9d6,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051644416 0} {<nil>} 3956684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946786816 0} {<nil>} 3854284Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:31 +0000 UTC,LastTransitionTime:2023-01-28 12:52:30 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:31 +0000 UTC,LastTransitionTime:2023-01-28 12:52:30 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:31 +0000 UTC,LastTransitionTime:2023-01-28 12:52:30 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 13:17:31 +0000 UTC,LastTransitionTime:2023-01-28 12:53:11 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.85,},NodeAddress{Type:ExternalIP,Address:13.212.207.67,},NodeAddress{Type:InternalDNS,Address:i-04736cc52ac74b9d6.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-04736cc52ac74b9d6.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-212-207-67.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec257e4128091cdda716cfd4093ff7c9,SystemUUID:ec257e41-2809-1cdd-a716-cfd4093ff7c9,BootID:86228385-d1a8-45e0-a965-764379f12219,KernelVersion:5.15.0-1028-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.25.6,KubeProxyVersion:v1.25.6,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.6],SizeBytes:63291593,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:99c0d6f1ad24a1aa1905d9c6534d193f268f7b23f9add2ae6bb41f31094bdd5c registry.k8s.io/e2e-test-images/nautilus:1.5],SizeBytes:49642095,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:f9c93b92b6ff750b41a93c4e4fe0bfe384597aeb841e2539d5444815c55b2d8f registry.k8s.io/e2e-test-images/sample-apiserver:1.17.5],SizeBytes:24316368,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:6e0546563b18872b0aa0cad7255a26bb9a87cb879b7fc3e2383c867ef4f706fb registry.k8s.io/sig-storage/csi-resizer:v1.3.0],SizeBytes:21671340,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-091a5c323546c884f kubernetes.io/csi/ebs.csi.aws.com^vol-0d582fe7c4d5e7b98],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-091a5c323546c884f,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0d582fe7c4d5e7b98,DevicePath:,},},Config:nil,},}
Jan 28 13:18:12.441: INFO: 
Logging kubelet events for node i-04736cc52ac74b9d6
Jan 28 13:18:12.659: INFO: 
Logging pods the kubelet thinks is on node i-04736cc52ac74b9d6
Jan 28 13:18:12.880: INFO: liveness-c7ea544d-b26a-4808-9bc3-4057a98e77f1 started at 2023-01-28 13:17:10 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:12.880: INFO: 	Container agnhost-container ready: true, restart count 2
Jan 28 13:18:12.880: INFO: ss2-0 started at 2023-01-28 13:18:07 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:12.880: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:12.880: INFO: netserver-2 started at 2023-01-28 13:12:48 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:12.880: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:12.880: INFO: service-proxy-disabled-p2pjj started at 2023-01-28 13:13:59 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:12.880: INFO: 	Container service-proxy-disabled ready: true, restart count 0
Jan 28 13:18:12.880: INFO: ss-0 started at 2023-01-28 13:16:45 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:12.880: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:12.880: INFO: webserver-54977698-ksdtb started at 2023-01-28 13:12:36 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:12.880: INFO: 	Container httpd ready: true, restart count 0
Jan 28 13:18:12.880: INFO: ss-0 started at 2023-01-28 13:18:11 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:12.880: INFO: 	Container webserver ready: false, restart count 0
Jan 28 13:18:12.880: INFO: cilium-6lzg7 started at 2023-01-28 12:52:31 +0000 UTC (1+1 container statuses recorded)
Jan 28 13:18:12.880: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 28 13:18:12.880: INFO: 	Container cilium-agent ready: true, restart count 3
Jan 28 13:18:12.880: INFO: ebs-csi-node-s9nrk started at 2023-01-28 12:52:31 +0000 UTC (0+3 container statuses recorded)
Jan 28 13:18:12.880: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 28 13:18:12.880: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 28 13:18:12.880: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 28 13:18:12.880: INFO: service-proxy-toggled-fslfq started at 2023-01-28 13:18:09 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:12.880: INFO: 	Container service-proxy-toggled ready: true, restart count 0
Jan 28 13:18:12.880: INFO: hostexec-i-04736cc52ac74b9d6-bk5pw started at 2023-01-28 13:18:08 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:12.880: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 28 13:18:12.880: INFO: netserver-2 started at 2023-01-28 13:14:01 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:12.880: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:13.613: INFO: 
Latency metrics for node i-04736cc52ac74b9d6
Jan 28 13:18:13.613: INFO: 
Logging node info for node i-08256d641436a507d
Jan 28 13:18:13.829: INFO: Node Info: &Node{ObjectMeta:{i-08256d641436a507d    72e91885-e899-4272-9caa-03bda484a486 39774 0 2023-01-28 12:50:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-08256d641436a507d kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-08256d641436a507d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-01-28 12:50:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-01-28 12:50:58 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-01-28 12:51:12 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 12:51:12 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-01-28 12:51:19 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-01-28 13:13:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-08256d641436a507d,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892264960 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787407360 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 13:13:25 +0000 UTC,LastTransitionTime:2023-01-28 12:50:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 13:13:25 +0000 UTC,LastTransitionTime:2023-01-28 12:50:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 13:13:25 +0000 UTC,LastTransitionTime:2023-01-28 12:50:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 13:13:25 +0000 UTC,LastTransitionTime:2023-01-28 12:51:19 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.46.131,},NodeAddress{Type:ExternalIP,Address:54.255.178.176,},NodeAddress{Type:InternalDNS,Address:i-08256d641436a507d.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-08256d641436a507d.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-255-178-176.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec23fc2ff8b73bac68ca7ff21819b359,SystemUUID:ec23fc2f-f8b7-3bac-68ca-7ff21819b359,BootID:a83d4d4c-9cad-4b57-b033-8eeba43e9638,KernelVersion:5.15.0-1028-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.25.6,KubeProxyVersion:v1.25.6,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:66a453db625abb268f4b3bbefc5a34a171d81e6e8796cecca54cfd71775c77c4 registry.k8s.io/etcdadm/etcd-manager:v3.0.20221209],SizeBytes:231502799,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.25.6],SizeBytes:129108435,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.25.6],SizeBytes:118450489,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.6],SizeBytes:63291593,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.25.6],SizeBytes:51931448,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-beta.2],SizeBytes:43830714,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-beta.2],SizeBytes:42821730,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:26802430,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:dcccdfba225e93ba2060a4c0b9072b50b0a564354c37bba6ed3ce89c326db58c registry.k8s.io/provider-aws/cloud-controller-manager:v1.25.2],SizeBytes:18280697,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-beta.2],SizeBytes:4965809,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Jan 28 13:18:13.830: INFO: 
Logging kubelet events for node i-08256d641436a507d
Jan 28 13:18:14.047: INFO: 
Logging pods the kubelet thinks is on node i-08256d641436a507d
Jan 28 13:18:14.275: INFO: etcd-manager-main-i-08256d641436a507d started at 2023-01-28 12:49:52 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:14.275: INFO: 	Container etcd-manager ready: true, restart count 0
Jan 28 13:18:14.275: INFO: kube-controller-manager-i-08256d641436a507d started at 2023-01-28 12:49:52 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:14.275: INFO: 	Container kube-controller-manager ready: true, restart count 2
Jan 28 13:18:14.275: INFO: kube-scheduler-i-08256d641436a507d started at 2023-01-28 12:49:52 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:14.275: INFO: 	Container kube-scheduler ready: true, restart count 0
Jan 28 13:18:14.275: INFO: ebs-csi-controller-749b596fb7-pltks started at 2023-01-28 12:50:58 +0000 UTC (0+5 container statuses recorded)
Jan 28 13:18:14.275: INFO: 	Container csi-attacher ready: true, restart count 0
Jan 28 13:18:14.275: INFO: 	Container csi-provisioner ready: true, restart count 0
Jan 28 13:18:14.275: INFO: 	Container csi-resizer ready: true, restart count 0
Jan 28 13:18:14.275: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 28 13:18:14.275: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 28 13:18:14.275: INFO: aws-cloud-controller-manager-qsphx started at 2023-01-28 12:50:58 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:14.275: INFO: 	Container aws-cloud-controller-manager ready: true, restart count 0
Jan 28 13:18:14.275: INFO: cilium-operator-674f5d6c65-x68tn started at 2023-01-28 12:51:00 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:14.275: INFO: 	Container cilium-operator ready: true, restart count 0
Jan 28 13:18:14.275: INFO: kops-controller-hh62m started at 2023-01-28 12:50:58 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:14.275: INFO: 	Container kops-controller ready: true, restart count 0
Jan 28 13:18:14.275: INFO: etcd-manager-events-i-08256d641436a507d started at 2023-01-28 12:49:52 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:14.275: INFO: 	Container etcd-manager ready: true, restart count 0
Jan 28 13:18:14.275: INFO: kube-apiserver-i-08256d641436a507d started at 2023-01-28 12:49:52 +0000 UTC (0+2 container statuses recorded)
Jan 28 13:18:14.275: INFO: 	Container healthcheck ready: true, restart count 0
Jan 28 13:18:14.275: INFO: 	Container kube-apiserver ready: true, restart count 1
Jan 28 13:18:14.275: INFO: etcd-manager-cilium-i-08256d641436a507d started at 2023-01-28 12:49:52 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:14.275: INFO: 	Container etcd-manager ready: true, restart count 0
Jan 28 13:18:14.275: INFO: ebs-csi-node-bj55j started at 2023-01-28 12:50:48 +0000 UTC (0+3 container statuses recorded)
Jan 28 13:18:14.275: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 28 13:18:14.275: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 28 13:18:14.275: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 28 13:18:14.275: INFO: cilium-sglkk started at 2023-01-28 12:50:58 +0000 UTC (1+1 container statuses recorded)
Jan 28 13:18:14.275: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 28 13:18:14.275: INFO: 	Container cilium-agent ready: true, restart count 1
Jan 28 13:18:14.275: INFO: dns-controller-56d4f686f6-5xr8q started at 2023-01-28 12:50:58 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:14.275: INFO: 	Container dns-controller ready: true, restart count 0
Jan 28 13:18:15.015: INFO: 
Latency metrics for node i-08256d641436a507d
Jan 28 13:18:15.015: INFO: 
Logging node info for node i-0cceed973097eb227
Jan 28 13:18:15.231: INFO: Node Info: &Node{ObjectMeta:{i-0cceed973097eb227    0d4efd12-5177-4438-95a1-cf4f7bc47cb0 45546 0 2023-01-28 12:52:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0cceed973097eb227 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0cceed973097eb227 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0cceed973097eb227"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-01-28 12:52:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-01-28 12:52:28 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-01-28 12:52:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-01-28 12:52:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-01-28 12:52:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-01-28 13:17:34 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-01-28 13:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0cceed973097eb227,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051652608 0} {<nil>} 3956692Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946795008 0} {<nil>} 3854292Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:39 +0000 UTC,LastTransitionTime:2023-01-28 12:52:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:39 +0000 UTC,LastTransitionTime:2023-01-28 12:52:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-01-28 13:17:39 +0000 UTC,LastTransitionTime:2023-01-28 12:52:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-01-28 13:17:39 +0000 UTC,LastTransitionTime:2023-01-28 12:52:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.50.150,},NodeAddress{Type:ExternalIP,Address:52.77.228.237,},NodeAddress{Type:InternalDNS,Address:i-0cceed973097eb227.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0cceed973097eb227.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-77-228-237.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bfb64b99756deba115b4f9f6a58b6,SystemUUID:ec2bfb64-b997-56de-ba11-5b4f9f6a58b6,BootID:104ee6bd-b21a-45d3-9eeb-f69e1fedb453,KernelVersion:5.15.0-1028-aws,OSImage:Ubuntu 22.04.1 LTS,ContainerRuntimeVersion:containerd://1.6.15,KubeletVersion:v1.25.6,KubeProxyVersion:v1.25.6,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:11e6a66017ba4e4b938c1612b7a54a3befcefd354796c04e1dba76873a13518e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.5],SizeBytes:112030526,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.25.6],SizeBytes:63291593,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:af7e3857d87770ddb40f5ea4f89b5a2709504ab1ee31f9ea4ab5823c045f2146 registry.k8s.io/e2e-test-images/agnhost:2.40],SizeBytes:51155161,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:20f25f275d46aa728f7615a1ccc19c78b2ed89435bf943a44b339f70f45508e6 registry.k8s.io/e2e-test-images/httpd:2.4.39-2],SizeBytes:41902010,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:1b9d1b2f36cb2dbee1960e82a9344aeb11bd4c4c03abf5e1853e0559c23855e3 registry.k8s.io/e2e-test-images/httpd:2.4.38-2],SizeBytes:40764680,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:6477988532358148d2e98f7c747db4e9250bbc7ad2664bf666348abf9ee1f5aa registry.k8s.io/sig-storage/csi-provisioner:v3.0.0],SizeBytes:22728994,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:89e900a160a986a1a7a4eba7f5259e510398fa87ca9b8a729e7dec59e04c7709 registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1],SizeBytes:22163966,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:80dec81b679a733fda448be92a2331150d99095947d04003ecff3dbd7f2a476a registry.k8s.io/sig-storage/csi-attacher:v3.3.0],SizeBytes:21444261,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:6029c252dae6178c99b580de72d7776158edbc81be0de15cedc4152a3acfed18 registry.k8s.io/sig-storage/hostpathplugin:v1.7.3],SizeBytes:15224494,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:4fd21f36075b44d1a423dfb262ad79202ce54e95f5cbc4622a6c1c38ab287ad6 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.0],SizeBytes:9132637,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.3.0],SizeBytes:8582494,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:55d0552eb6538050ea7741e46b35d27eccffeeaed7010f9f2bad0a89c149bc6f registry.k8s.io/e2e-test-images/nginx:1.15-2],SizeBytes:7000509,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 registry.k8s.io/e2e-test-images/nginx:1.14-2],SizeBytes:6979041,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-02b16e7f805809f6d kubernetes.io/csi/ebs.csi.aws.com^vol-0d793e6008e647344],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0d793e6008e647344,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02b16e7f805809f6d,DevicePath:,},},Config:nil,},}
Jan 28 13:18:15.231: INFO: 
Logging kubelet events for node i-0cceed973097eb227
Jan 28 13:18:15.449: INFO: 
Logging pods the kubelet thinks is on node i-0cceed973097eb227
Jan 28 13:18:15.670: INFO: coredns-85d58b74c8-65c7c started at 2023-01-28 12:52:48 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:15.670: INFO: 	Container coredns ready: true, restart count 0
Jan 28 13:18:15.670: INFO: ss-2 started at 2023-01-28 13:17:31 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:15.670: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:15.670: INFO: ss2-1 started at 2023-01-28 13:18:05 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:15.670: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:15.670: INFO: verify-service-up-host-exec-pod started at 2023-01-28 13:18:12 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:15.670: INFO: 	Container agnhost-container ready: true, restart count 0
Jan 28 13:18:15.670: INFO: coredns-autoscaler-5b9dc8bb99-t6qm5 started at 2023-01-28 12:52:48 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:15.670: INFO: 	Container autoscaler ready: true, restart count 0
Jan 28 13:18:15.670: INFO: webserver-54977698-hmpbz started at 2023-01-28 13:12:42 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:15.670: INFO: 	Container httpd ready: true, restart count 0
Jan 28 13:18:15.670: INFO: webserver-54977698-zgr7q started at 2023-01-28 13:12:54 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:15.670: INFO: 	Container httpd ready: true, restart count 0
Jan 28 13:18:15.670: INFO: ebs-csi-node-m7vpr started at 2023-01-28 12:52:29 +0000 UTC (0+3 container statuses recorded)
Jan 28 13:18:15.670: INFO: 	Container ebs-plugin ready: true, restart count 0
Jan 28 13:18:15.670: INFO: 	Container liveness-probe ready: true, restart count 0
Jan 28 13:18:15.670: INFO: 	Container node-driver-registrar ready: true, restart count 0
Jan 28 13:18:15.670: INFO: service-proxy-toggled-s2q7w started at 2023-01-28 13:18:09 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:15.670: INFO: 	Container service-proxy-toggled ready: true, restart count 0
Jan 28 13:18:15.670: INFO: pod-subpath-test-secret-gdwk started at 2023-01-28 13:17:58 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:15.670: INFO: 	Container test-container-subpath-secret-gdwk ready: true, restart count 0
Jan 28 13:18:15.670: INFO: cilium-j76rx started at 2023-01-28 12:52:29 +0000 UTC (1+1 container statuses recorded)
Jan 28 13:18:15.670: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Jan 28 13:18:15.670: INFO: 	Container cilium-agent ready: true, restart count 3
Jan 28 13:18:15.670: INFO: netserver-3 started at 2023-01-28 13:14:02 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:15.670: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:15.670: INFO: ss-2 started at 2023-01-28 13:15:35 +0000 UTC (0+1 container statuses recorded)
Jan 28 13:18:15.670: INFO: 	Container webserver ready: true, restart count 0
Jan 28 13:18:16.459: INFO: 
Latency metrics for node i-0cceed973097eb227
Jan 28 13:18:16.459: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
�[1mSTEP:�[0m Destroying namespace "provisioning-2881" for this suite. �[38;5;243m01/28/23 13:18:16.675�[0m

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

kubetest2 Test 28m20s

exit status 255
				from junit_runner.xml

Filter through log files | View test history on testgrid

Show 869 Passed Tests

Show 6513 Skipped Tests

Result	FAILURE
Tests	4 failed / 869 succeeded
Started	2023-01-28 12:46
Elapsed	42m17s
Revision	master