e2e-kops-aws-cni-cilium-etcd #1634402420526157824 Results

Test Failures

Kubernetes e2e suite [It] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for multiple CRDs of same group and version but different kinds [Conformance] 1m25s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-api\-machinery\]\sCustomResourcePublishOpenAPI\s\[Privileged\:ClusterAdmin\]\sworks\sfor\smultiple\sCRDs\sof\ssame\sgroup\sand\sversion\sbut\sdifferent\skinds\s\[Conformance\]$'

test/e2e/apimachinery/crd_publish_openapi.go:374
k8s.io/kubernetes/test/e2e/apimachinery.glob..func6.7()
	test/e2e/apimachinery/crd_publish_openapi.go:374 +0x34b
				[BeforeEach] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin]
  set up framework | framework.go:178
�[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m03/11/23 04:29:26.465�[0m
Mar 11 04:29:26.465: INFO: >>> kubeConfig: /root/.kube/config
�[1mSTEP:�[0m Building a namespace api object, basename crd-publish-openapi �[38;5;243m03/11/23 04:29:26.466�[0m
�[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m03/11/23 04:29:27.123�[0m
�[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m03/11/23 04:29:27.551�[0m
[BeforeEach] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin]
  test/e2e/framework/metrics/init/init.go:31
[It] works for multiple CRDs of same group and version but different kinds [Conformance]
  test/e2e/apimachinery/crd_publish_openapi.go:357
�[1mSTEP:�[0m CRs in the same group and version but different kinds (two CRDs) show up in OpenAPI documentation �[38;5;243m03/11/23 04:29:27.981�[0m
Mar 11 04:29:27.982: INFO: >>> kubeConfig: /root/.kube/config
Mar 11 04:29:42.880: INFO: >>> kubeConfig: /root/.kube/config
Mar 11 04:30:42.479: FAIL: failed to wait for definition "com.example.crd-publish-openapi-test-common-group.v6.e2e-test-crd-publish-openapi-9980-crd" to be served with the right OpenAPI schema: failed to wait for OpenAPI spec validating condition: Get "https://api.e2e-e2e-kops-aws-cni-cilium-etcd.test-cncf-aws.k8s.io/openapi/v2": dial tcp 54.255.194.65:443: i/o timeout; lastMsg: 

Full Stack Trace
k8s.io/kubernetes/test/e2e/apimachinery.glob..func6.7()
	test/e2e/apimachinery/crd_publish_openapi.go:374 +0x34b
[AfterEach] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin]
  test/e2e/framework/node/init/init.go:32
Mar 11 04:30:42.479: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[DeferCleanup (Each)] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin]
  test/e2e/framework/metrics/init/init.go:33
[DeferCleanup (Each)] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin]
  dump namespaces | framework.go:196
�[1mSTEP:�[0m dump namespace information after failure �[38;5;243m03/11/23 04:30:43.568�[0m
�[1mSTEP:�[0m Collecting events from namespace "crd-publish-openapi-9117". �[38;5;243m03/11/23 04:30:43.568�[0m
�[1mSTEP:�[0m Found 0 events. �[38;5;243m03/11/23 04:30:43.783�[0m
Mar 11 04:30:43.998: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Mar 11 04:30:43.998: INFO: 
Mar 11 04:30:44.214: INFO: 
Logging node info for node i-0011f85bb22127167
Mar 11 04:30:44.429: INFO: Node Info: &Node{ObjectMeta:{i-0011f85bb22127167    d4f3e0f9-f7c8-4aac-b938-1997d491f8e8 50749 0 2023-03-11 04:04:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0011f85bb22127167 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0011f85bb22127167 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0011f85bb22127167"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 04:04:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:47 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 04:04:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 04:04:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 04:04:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-11 04:30:07 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-11 04:30:09 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0011f85bb22127167,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:09 +0000 UTC,LastTransitionTime:2023-03-11 04:04:47 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:09 +0000 UTC,LastTransitionTime:2023-03-11 04:04:47 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:09 +0000 UTC,LastTransitionTime:2023-03-11 04:04:47 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 04:30:09 +0000 UTC,LastTransitionTime:2023-03-11 04:05:02 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.97,},NodeAddress{Type:ExternalIP,Address:54.254.3.232,},NodeAddress{Type:InternalDNS,Address:i-0011f85bb22127167.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0011f85bb22127167.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-3-232.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2cf42c34dc130918d57386d1dc0cf7,SystemUUID:ec2cf42c-34dc-1309-18d5-7386d1dc0cf7,BootID:1bccc978-bb3f-4f8e-87dd-09d834c280d3,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.26.2,KubeProxyVersion:v1.26.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.2],SizeBytes:67213512,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-04aa86a7d77b880a9],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-04aa86a7d77b880a9,DevicePath:,},},Config:nil,},}
Mar 11 04:30:44.429: INFO: 
Logging kubelet events for node i-0011f85bb22127167
Mar 11 04:30:44.655: INFO: 
Logging pods the kubelet thinks is on node i-0011f85bb22127167
Mar 11 04:30:44.874: INFO: coredns-559769c974-mcnnl started at 2023-03-11 04:05:20 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:30:44.874: INFO: 	Container coredns ready: true, restart count 0
Mar 11 04:30:44.874: INFO: ebs-csi-node-dr4gl started at 2023-03-11 04:04:47 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:30:44.874: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:30:44.874: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:30:44.874: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:30:44.874: INFO: ss-1 started at 2023-03-11 04:30:05 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:30:44.874: INFO: 	Container webserver ready: true, restart count 0
Mar 11 04:30:44.874: INFO: cilium-z4lzr started at 2023-03-11 04:04:47 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:30:44.874: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 04:30:44.874: INFO: 	Container cilium-agent ready: true, restart count 3
Mar 11 04:30:45.674: INFO: 
Latency metrics for node i-0011f85bb22127167
Mar 11 04:30:45.674: INFO: 
Logging node info for node i-0329df98c53e1cce2
Mar 11 04:30:46.046: INFO: Node Info: &Node{ObjectMeta:{i-0329df98c53e1cce2    18579d8f-b168-49a3-ad57-ba94348535ca 50809 0 2023-03-11 04:04:36 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0329df98c53e1cce2 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0329df98c53e1cce2 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0329df98c53e1cce2"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-03-11 04:04:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 04:04:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:37 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 04:04:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-11 04:30:19 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0329df98c53e1cce2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:19 +0000 UTC,LastTransitionTime:2023-03-11 04:04:36 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:19 +0000 UTC,LastTransitionTime:2023-03-11 04:04:36 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:19 +0000 UTC,LastTransitionTime:2023-03-11 04:04:36 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 04:30:19 +0000 UTC,LastTransitionTime:2023-03-11 04:04:53 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.134,},NodeAddress{Type:ExternalIP,Address:54.254.230.138,},NodeAddress{Type:InternalDNS,Address:i-0329df98c53e1cce2.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0329df98c53e1cce2.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-230-138.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2707505e56c78d5055fa8514534cec,SystemUUID:ec270750-5e56-c78d-5055-fa8514534cec,BootID:e848eeb0-78cc-41f1-993e-ef001dbf3b82,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.26.2,KubeProxyVersion:v1.26.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.2],SizeBytes:67213512,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 04:30:46.046: INFO: 
Logging kubelet events for node i-0329df98c53e1cce2
Mar 11 04:30:46.265: INFO: 
Logging pods the kubelet thinks is on node i-0329df98c53e1cce2
Mar 11 04:30:46.486: INFO: cilium-kqgwb started at 2023-03-11 04:04:37 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:30:46.486: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 04:30:46.486: INFO: 	Container cilium-agent ready: true, restart count 4
Mar 11 04:30:46.486: INFO: ebs-csi-controller-85b78cc797-gcf7r started at 2023-03-11 04:04:54 +0000 UTC (0+5 container statuses recorded)
Mar 11 04:30:46.486: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 04:30:46.486: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 04:30:46.486: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 04:30:46.486: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:30:46.486: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:30:46.486: INFO: ebs-csi-node-mt78m started at 2023-03-11 04:04:37 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:30:46.486: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:30:46.486: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:30:46.486: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:30:47.227: INFO: 
Latency metrics for node i-0329df98c53e1cce2
Mar 11 04:30:47.227: INFO: 
Logging node info for node i-0741369cdbade9399
Mar 11 04:30:47.442: INFO: Node Info: &Node{ObjectMeta:{i-0741369cdbade9399    bf0e7ac5-1ef1-4825-a6a5-331297cd61eb 50856 0 2023-03-11 04:04:40 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0741369cdbade9399 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0741369cdbade9399 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0741369cdbade9399"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 04:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:40 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 04:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 04:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 04:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-03-11 04:30:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 04:30:29 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0741369cdbade9399,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:12 +0000 UTC,LastTransitionTime:2023-03-11 04:04:40 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:12 +0000 UTC,LastTransitionTime:2023-03-11 04:04:40 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:12 +0000 UTC,LastTransitionTime:2023-03-11 04:04:40 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 04:30:12 +0000 UTC,LastTransitionTime:2023-03-11 04:04:56 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.36.32,},NodeAddress{Type:ExternalIP,Address:52.221.224.194,},NodeAddress{Type:InternalDNS,Address:i-0741369cdbade9399.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0741369cdbade9399.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-224-194.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2227001f7fcddc5c884cabbabf0e0b,SystemUUID:ec222700-1f7f-cddc-5c88-4cabbabf0e0b,BootID:e32683e2-f725-4230-a02d-861048b9a290,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.26.2,KubeProxyVersion:v1.26.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.2],SizeBytes:67213512,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0f6a3bb5ff6226434],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0f6a3bb5ff6226434,DevicePath:,},},Config:nil,},}
Mar 11 04:30:47.443: INFO: 
Logging kubelet events for node i-0741369cdbade9399
Mar 11 04:30:47.661: INFO: 
Logging pods the kubelet thinks is on node i-0741369cdbade9399
Mar 11 04:30:47.879: INFO: ebs-csi-node-9z96v started at 2023-03-11 04:04:41 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:30:47.879: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:30:47.879: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:30:47.879: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:30:47.879: INFO: ss-0 started at 2023-03-11 04:30:11 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:30:47.879: INFO: 	Container webserver ready: true, restart count 0
Mar 11 04:30:47.879: INFO: cilium-gvqjv started at 2023-03-11 04:04:41 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:30:47.879: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 04:30:47.879: INFO: 	Container cilium-agent ready: true, restart count 3
Mar 11 04:30:47.879: INFO: funny-ip-bmnbw started at 2023-03-11 04:28:26 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:30:47.879: INFO: 	Container funny-ip ready: true, restart count 0
Mar 11 04:30:48.605: INFO: 
Latency metrics for node i-0741369cdbade9399
Mar 11 04:30:48.605: INFO: 
Logging node info for node i-08076ffd11575a948
Mar 11 04:30:48.820: INFO: Node Info: &Node{ObjectMeta:{i-08076ffd11575a948    eb67a4db-a950-406a-a15b-0d0249beb2f5 50949 0 2023-03-11 04:02:21 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-08076ffd11575a948 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-08076ffd11575a948"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 04:02:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 04:02:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:03:06 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:03:06 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 04:03:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 04:30:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-08076ffd11575a948,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:45 +0000 UTC,LastTransitionTime:2023-03-11 04:02:19 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:45 +0000 UTC,LastTransitionTime:2023-03-11 04:02:19 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:45 +0000 UTC,LastTransitionTime:2023-03-11 04:02:19 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 04:30:45 +0000 UTC,LastTransitionTime:2023-03-11 04:03:22 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.32.13,},NodeAddress{Type:ExternalIP,Address:54.255.194.65,},NodeAddress{Type:InternalDNS,Address:i-08076ffd11575a948.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-08076ffd11575a948.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-255-194-65.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec28e5c7b8aa64aa5652371517e966ea,SystemUUID:ec28e5c7-b8aa-64aa-5652-371517e966ea,BootID:bcab411b-a8a1-415e-8bc4-977b594974ab,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.26.2,KubeProxyVersion:v1.26.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.26.2],SizeBytes:135195085,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.26.2],SizeBytes:124561719,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.2],SizeBytes:67213512,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.26.2],SizeBytes:57669942,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45226387,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43628953,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:26802430,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074660,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 04:30:48.820: INFO: 
Logging kubelet events for node i-08076ffd11575a948
Mar 11 04:30:49.039: INFO: 
Logging pods the kubelet thinks is on node i-08076ffd11575a948
Mar 11 04:30:49.480: INFO: etcd-manager-events-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (11+1 container statuses recorded)
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 11 04:30:49.480: INFO: etcd-manager-main-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (11+1 container statuses recorded)
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 11 04:30:49.480: INFO: ebs-csi-node-wtm58 started at 2023-03-11 04:03:00 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:30:49.480: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:30:49.480: INFO: cilium-d2jhz started at 2023-03-11 04:03:00 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:30:49.480: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 11 04:30:49.480: INFO: dns-controller-bdb55cccc-wrj9v started at 2023-03-11 04:03:00 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:30:49.480: INFO: 	Container dns-controller ready: true, restart count 0
Mar 11 04:30:49.480: INFO: cilium-operator-69bd9b65dc-2bdrx started at 2023-03-11 04:03:00 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:30:49.480: INFO: 	Container cilium-operator ready: true, restart count 1
Mar 11 04:30:49.480: INFO: etcd-manager-cilium-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (11+1 container statuses recorded)
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 11 04:30:49.480: INFO: kube-apiserver-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (0+2 container statuses recorded)
Mar 11 04:30:49.480: INFO: 	Container healthcheck ready: true, restart count 0
Mar 11 04:30:49.480: INFO: 	Container kube-apiserver ready: true, restart count 3
Mar 11 04:30:49.480: INFO: kube-controller-manager-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:30:49.480: INFO: 	Container kube-controller-manager ready: true, restart count 4
Mar 11 04:30:49.480: INFO: kube-scheduler-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:30:49.480: INFO: 	Container kube-scheduler ready: true, restart count 0
Mar 11 04:30:49.480: INFO: kops-controller-msq8z started at 2023-03-11 04:03:00 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:30:49.480: INFO: 	Container kops-controller ready: true, restart count 0
Mar 11 04:30:49.480: INFO: aws-cloud-controller-manager-7spkx started at 2023-03-11 04:03:00 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:30:49.480: INFO: 	Container aws-cloud-controller-manager ready: true, restart count 0
Mar 11 04:30:50.182: INFO: 
Latency metrics for node i-08076ffd11575a948
Mar 11 04:30:50.182: INFO: 
Logging node info for node i-0ceb4774ac2f61618
Mar 11 04:30:50.398: INFO: Node Info: &Node{ObjectMeta:{i-0ceb4774ac2f61618    e54bc4e4-c393-4ff9-8459-fae12254c541 50734 0 2023-03-11 04:04:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0ceb4774ac2f61618 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0ceb4774ac2f61618 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0ceb4774ac2f61618"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 04:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:36 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:36 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 04:04:36 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 04:04:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 04:30:08 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0ceb4774ac2f61618,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 04:29:58 +0000 UTC,LastTransitionTime:2023-03-11 04:04:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 04:29:58 +0000 UTC,LastTransitionTime:2023-03-11 04:04:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 04:29:58 +0000 UTC,LastTransitionTime:2023-03-11 04:04:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 04:29:58 +0000 UTC,LastTransitionTime:2023-03-11 04:04:51 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.15,},NodeAddress{Type:ExternalIP,Address:13.229.94.195,},NodeAddress{Type:InternalDNS,Address:i-0ceb4774ac2f61618.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ceb4774ac2f61618.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-229-94-195.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2b5f3d9b64f14ea7e43e834dad161f,SystemUUID:ec2b5f3d-9b64-f14e-a7e4-3e834dad161f,BootID:a024bb6c-a740-4539-b41b-e60585607279,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.26.2,KubeProxyVersion:v1.26.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.2],SizeBytes:67213512,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 04:30:50.398: INFO: 
Logging kubelet events for node i-0ceb4774ac2f61618
Mar 11 04:30:50.617: INFO: 
Logging pods the kubelet thinks is on node i-0ceb4774ac2f61618
Mar 11 04:30:50.837: INFO: coredns-autoscaler-7cb5c5b969-wrq84 started at 2023-03-11 04:04:51 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:30:50.837: INFO: 	Container autoscaler ready: true, restart count 0
Mar 11 04:30:50.837: INFO: ebs-csi-node-l4jm2 started at 2023-03-11 04:04:36 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:30:50.837: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:30:50.837: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:30:50.837: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:30:50.837: INFO: cilium-9vxfg started at 2023-03-11 04:04:36 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:30:50.837: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 04:30:50.837: INFO: 	Container cilium-agent ready: true, restart count 3
Mar 11 04:30:50.837: INFO: execpodflzqr started at 2023-03-11 04:28:35 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:30:50.837: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 04:30:50.837: INFO: ebs-csi-controller-85b78cc797-xlv4h started at 2023-03-11 04:04:51 +0000 UTC (0+5 container statuses recorded)
Mar 11 04:30:50.837: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 04:30:50.837: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 04:30:50.837: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 04:30:50.837: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:30:50.837: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:30:50.837: INFO: coredns-559769c974-xdqkc started at 2023-03-11 04:04:51 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:30:50.837: INFO: 	Container coredns ready: true, restart count 0
Mar 11 04:30:51.564: INFO: 
Latency metrics for node i-0ceb4774ac2f61618
[DeferCleanup (Each)] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin]
  tear down framework | framework.go:193
�[1mSTEP:�[0m Destroying namespace "crd-publish-openapi-9117" for this suite. �[38;5;243m03/11/23 04:30:51.564�[0m

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

Kubernetes e2e suite [It] [sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should not deadlock when a pod's predecessor fails 3m31s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-apps\]\sStatefulSet\sBasic\sStatefulSet\sfunctionality\s\[StatefulSetBasic\]\sshould\snot\sdeadlock\swhen\sa\spod\'s\spredecessor\sfails$'

test/e2e/framework/statefulset/rest.go:69
k8s.io/kubernetes/test/e2e/framework/statefulset.GetPodList({0x8023d88, 0xc002f241a0}, 0xc001758500)
	test/e2e/framework/statefulset/rest.go:69 +0x153
k8s.io/kubernetes/test/e2e/framework/statefulset.WaitForRunning.func1()
	test/e2e/framework/statefulset/wait.go:37 +0x4a
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1({0x27448b1, 0x0})
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:222 +0x1b
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x7fe6a48?, 0xc000084048?}, 0xc0043d8db0?)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:235 +0x57
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.WaitForWithContext({0x7fe6a48, 0xc000084048}, 0xc004456978, 0x2fdde6a?)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:662 +0x10c
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7fe6a48, 0xc000084048}, 0x28?, 0x2fdca05?, 0x20?)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:596 +0x9a
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext({0x7fe6a48, 0xc000084048}, 0x65d0300?, 0xc00182be78?, 0x262c967?)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:528 +0x4a
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollImmediate(0x120?, 0x0?, 0x0?)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:514 +0x50
k8s.io/kubernetes/test/e2e/framework/statefulset.WaitForRunning({0x8023d88?, 0xc002f241a0}, 0x2, 0x2, 0xc001758500)
	test/e2e/framework/statefulset/wait.go:35 +0xbd
k8s.io/kubernetes/test/e2e/framework/statefulset.WaitForRunningAndReady(...)
	test/e2e/framework/statefulset/wait.go:80
k8s.io/kubernetes/test/e2e/apps.glob..func10.2.5()
	test/e2e/apps/statefulset.go:289 +0x326
				[BeforeEach] [sig-apps] StatefulSet
  set up framework | framework.go:178
�[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m03/11/23 04:29:22.75�[0m
Mar 11 04:29:22.750: INFO: >>> kubeConfig: /root/.kube/config
�[1mSTEP:�[0m Building a namespace api object, basename statefulset �[38;5;243m03/11/23 04:29:22.751�[0m
�[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m03/11/23 04:29:29.284�[0m
�[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m03/11/23 04:29:33.626�[0m
[BeforeEach] [sig-apps] StatefulSet
  test/e2e/framework/metrics/init/init.go:31
[BeforeEach] [sig-apps] StatefulSet
  test/e2e/apps/statefulset.go:98
[BeforeEach] Basic StatefulSet functionality [StatefulSetBasic]
  test/e2e/apps/statefulset.go:113
�[1mSTEP:�[0m Creating service test in namespace statefulset-6074 �[38;5;243m03/11/23 04:29:38.017�[0m
[It] should not deadlock when a pod's predecessor fails
  test/e2e/apps/statefulset.go:258
�[1mSTEP:�[0m Creating statefulset ss in namespace statefulset-6074 �[38;5;243m03/11/23 04:29:40.114�[0m
Mar 11 04:29:42.191: INFO: Default storage class: "kops-csi-1-21"
Mar 11 04:29:46.398: INFO: Waiting for pod ss-0 to enter Running - Ready=false, currently Pending - Ready=false
Mar 11 04:29:59.042: INFO: Waiting for pod ss-0 to enter Running - Ready=false, currently Running - Ready=false
�[1mSTEP:�[0m Resuming stateful pod at index 0. �[38;5;243m03/11/23 04:29:59.042�[0m
Mar 11 04:29:59.258: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/71529c8c-bfc0-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-cni-cilium-etcd.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=statefulset-6074 exec ss-0 -- /bin/sh -x -c dd if=/dev/zero of=/data/statefulset-continue bs=1 count=1 conv=fsync'
Mar 11 04:30:01.359: INFO: stderr: "+ dd 'if=/dev/zero' 'of=/data/statefulset-continue' 'bs=1' 'count=1' 'conv=fsync'\n1+0 records in\n1+0 records out\n"
Mar 11 04:30:01.359: INFO: stdout: ""
Mar 11 04:30:01.359: INFO: Resumed pod ss-0
�[1mSTEP:�[0m Waiting for stateful pod at index 1 to enter running. �[38;5;243m03/11/23 04:30:01.359�[0m
Mar 11 04:30:01.575: INFO: Waiting for pod ss-0 to enter Running - Ready=true, currently Running - Ready=true
Mar 11 04:30:01.575: INFO: Waiting for pod ss-1 to enter Running - Ready=false, currently Pending - Ready=false
Mar 11 04:30:11.791: INFO: Waiting for pod ss-0 to enter Running - Ready=true, currently Running - Ready=true
Mar 11 04:30:11.791: INFO: Waiting for pod ss-1 to enter Running - Ready=false, currently Running - Ready=false
�[1mSTEP:�[0m Deleting healthy stateful pod at index 0. �[38;5;243m03/11/23 04:30:11.791�[0m
�[1mSTEP:�[0m Confirming stateful pod at index 0 is recreated. �[38;5;243m03/11/23 04:30:12.02�[0m
Mar 11 04:30:12.236: INFO: Waiting for pod ss-0 to enter Running - Ready=true, currently Pending - Ready=false
Mar 11 04:30:22.453: INFO: Waiting for pod ss-0 to enter Running - Ready=true, currently Pending - Ready=false
Mar 11 04:30:32.453: INFO: Waiting for pod ss-0 to enter Running - Ready=true, currently Running - Ready=true
Mar 11 04:30:32.453: INFO: Waiting for pod ss-1 to enter Running - Ready=false, currently Running - Ready=false
�[1mSTEP:�[0m Resuming stateful pod at index 1. �[38;5;243m03/11/23 04:30:32.453�[0m
Mar 11 04:30:32.668: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/71529c8c-bfc0-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-cni-cilium-etcd.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=statefulset-6074 exec ss-1 -- /bin/sh -x -c dd if=/dev/zero of=/data/statefulset-continue bs=1 count=1 conv=fsync'
Mar 11 04:30:34.797: INFO: stderr: "+ dd 'if=/dev/zero' 'of=/data/statefulset-continue' 'bs=1' 'count=1' 'conv=fsync'\n1+0 records in\n1+0 records out\n"
Mar 11 04:30:34.798: INFO: stdout: ""
Mar 11 04:30:34.798: INFO: Resumed pod ss-1
�[1mSTEP:�[0m Confirming all stateful pods in statefulset are created. �[38;5;243m03/11/23 04:30:34.798�[0m
Mar 11 04:30:35.014: INFO: Waiting for pod ss-0 to enter Running - Ready=true, currently Running - Ready=true
Mar 11 04:30:35.014: INFO: Waiting for pod ss-1 to enter Running - Ready=true, currently Running - Ready=false
Mar 11 04:31:20.015: INFO: Unexpected error: 
    <*url.Error | 0xc0043ca000>: {
        Op: "Get",
        URL: "https://api.e2e-e2e-kops-aws-cni-cilium-etcd.test-cncf-aws.k8s.io/api/v1/namespaces/statefulset-6074/pods?labelSelector=baz%3Dblah%2Cfoo%3Dbar",
        Err: <*errors.errorString | 0xc0001e2150>{
            s: "http2: client connection lost",
        },
    }
Mar 11 04:31:20.015: FAIL: Get "https://api.e2e-e2e-kops-aws-cni-cilium-etcd.test-cncf-aws.k8s.io/api/v1/namespaces/statefulset-6074/pods?labelSelector=baz%3Dblah%2Cfoo%3Dbar": http2: client connection lost

Full Stack Trace
k8s.io/kubernetes/test/e2e/framework/statefulset.GetPodList({0x8023d88, 0xc002f241a0}, 0xc001758500)
	test/e2e/framework/statefulset/rest.go:69 +0x153
k8s.io/kubernetes/test/e2e/framework/statefulset.WaitForRunning.func1()
	test/e2e/framework/statefulset/wait.go:37 +0x4a
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1({0x27448b1, 0x0})
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:222 +0x1b
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x7fe6a48?, 0xc000084048?}, 0xc0043d8db0?)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:235 +0x57
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.WaitForWithContext({0x7fe6a48, 0xc000084048}, 0xc004456978, 0x2fdde6a?)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:662 +0x10c
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7fe6a48, 0xc000084048}, 0x28?, 0x2fdca05?, 0x20?)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:596 +0x9a
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext({0x7fe6a48, 0xc000084048}, 0x65d0300?, 0xc00182be78?, 0x262c967?)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:528 +0x4a
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollImmediate(0x120?, 0x0?, 0x0?)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:514 +0x50
k8s.io/kubernetes/test/e2e/framework/statefulset.WaitForRunning({0x8023d88?, 0xc002f241a0}, 0x2, 0x2, 0xc001758500)
	test/e2e/framework/statefulset/wait.go:35 +0xbd
k8s.io/kubernetes/test/e2e/framework/statefulset.WaitForRunningAndReady(...)
	test/e2e/framework/statefulset/wait.go:80
k8s.io/kubernetes/test/e2e/apps.glob..func10.2.5()
	test/e2e/apps/statefulset.go:289 +0x326
E0311 04:31:20.015615    7677 runtime.go:79] Observed a panic: types.GinkgoError{Heading:"Your Test Panicked", Message:"When you, or your assertion library, calls Ginkgo's Fail(),\nGinkgo panics to prevent subsequent assertions from running.\n\nNormally Ginkgo rescues this panic so you shouldn't see it.\n\nHowever, if you make an assertion in a goroutine, Ginkgo can't capture the panic.\nTo circumvent this, you should call\n\n\tdefer GinkgoRecover()\n\nat the top of the goroutine that caused this panic.\n\nAlternatively, you may have made an assertion outside of a Ginkgo\nleaf node (e.g. in a container node or some out-of-band function) - please move your assertion to\nan appropriate Ginkgo node (e.g. a BeforeSuite, BeforeEach, It, etc...).", DocLink:"mental-model-how-ginkgo-handles-failure", CodeLocation:types.CodeLocation{FileName:"test/e2e/framework/statefulset/rest.go", LineNumber:69, FullStackTrace:"k8s.io/kubernetes/test/e2e/framework/statefulset.GetPodList({0x8023d88, 0xc002f241a0}, 0xc001758500)\n\ttest/e2e/framework/statefulset/rest.go:69 +0x153\nk8s.io/kubernetes/test/e2e/framework/statefulset.WaitForRunning.func1()\n\ttest/e2e/framework/statefulset/wait.go:37 +0x4a\nk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1({0x27448b1, 0x0})\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:222 +0x1b\nk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x7fe6a48?, 0xc000084048?}, 0xc0043d8db0?)\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:235 +0x57\nk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.WaitForWithContext({0x7fe6a48, 0xc000084048}, 0xc004456978, 0x2fdde6a?)\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:662 +0x10c\nk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7fe6a48, 0xc000084048}, 0x28?, 0x2fdca05?, 0x20?)\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:596 +0x9a\nk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext({0x7fe6a48, 0xc000084048}, 0x65d0300?, 0xc00182be78?, 0x262c967?)\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:528 +0x4a\nk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollImmediate(0x120?, 0x0?, 0x0?)\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:514 +0x50\nk8s.io/kubernetes/test/e2e/framework/statefulset.WaitForRunning({0x8023d88?, 0xc002f241a0}, 0x2, 0x2, 0xc001758500)\n\ttest/e2e/framework/statefulset/wait.go:35 +0xbd\nk8s.io/kubernetes/test/e2e/framework/statefulset.WaitForRunningAndReady(...)\n\ttest/e2e/framework/statefulset/wait.go:80\nk8s.io/kubernetes/test/e2e/apps.glob..func10.2.5()\n\ttest/e2e/apps/statefulset.go:289 +0x326", CustomMessage:""}} (�[1m�[38;5;9mYour Test Panicked�[0m
�[38;5;243mtest/e2e/framework/statefulset/rest.go:69�[0m
  When you, or your assertion library, calls Ginkgo's Fail(),
  Ginkgo panics to prevent subsequent assertions from running.

  Normally Ginkgo rescues this panic so you shouldn't see it.

  However, if you make an assertion in a goroutine, Ginkgo can't capture the
  panic.
  To circumvent this, you should call

  	defer GinkgoRecover()

  at the top of the goroutine that caused this panic.

  Alternatively, you may have made an assertion outside of a Ginkgo
  leaf node (e.g. in a container node or some out-of-band function) - please
  move your assertion to
  an appropriate Ginkgo node (e.g. a BeforeSuite, BeforeEach, It, etc...).

  �[1mLearn more at:�[0m
  �[38;5;14m�[4mhttp://onsi.github.io/ginkgo/#mental-model-how-ginkgo-handles-failure�[0m
)
goroutine 5685 [running]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic({0x70f0660?, 0xc000168620})
	vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000168620?})
	vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
panic({0x70f0660, 0xc000168620})
	/usr/local/go/src/runtime/panic.go:884 +0x212
k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2.Fail({0xc0011ec0d0, 0xc8}, {0xc00182b5d8?, 0x75ba91a?, 0xc00182b5f8?})
	vendor/github.com/onsi/ginkgo/v2/core_dsl.go:352 +0x225
k8s.io/kubernetes/test/e2e/framework.Fail({0xc0032a8000, 0xb3}, {0xc00182b670?, 0xc0032a8000?, 0xc00182b698?})
	test/e2e/framework/log.go:61 +0x145
k8s.io/kubernetes/test/e2e/framework.ExpectNoErrorWithOffset(0x1, {0x7fb3e40, 0xc0043ca000}, {0x0?, 0xc00456bc30?, 0x10?})
	test/e2e/framework/expect.go:76 +0x267
k8s.io/kubernetes/test/e2e/framework.ExpectNoError(...)
	test/e2e/framework/expect.go:43
k8s.io/kubernetes/test/e2e/framework/statefulset.GetPodList({0x8023d88, 0xc002f241a0}, 0xc001758500)
	test/e2e/framework/statefulset/rest.go:69 +0x153
k8s.io/kubernetes/test/e2e/framework/statefulset.WaitForRunning.func1()
	test/e2e/framework/statefulset/wait.go:37 +0x4a
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1({0x27448b1, 0x0})
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:222 +0x1b
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x7fe6a48?, 0xc000084048?}, 0xc0043d8db0?)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:235 +0x57
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.WaitForWithContext({0x7fe6a48, 0xc000084048}, 0xc004456978, 0x2fdde6a?)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:662 +0x10c
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll({0x7fe6a48, 0xc000084048}, 0x28?, 0x2fdca05?, 0x20?)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:596 +0x9a
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext({0x7fe6a48, 0xc000084048}, 0x65d0300?, 0xc00182be78?, 0x262c967?)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:528 +0x4a
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.PollImmediate(0x120?, 0x0?, 0x0?)
	vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:514 +0x50
k8s.io/kubernetes/test/e2e/framework/statefulset.WaitForRunning({0x8023d88?, 0xc002f241a0}, 0x2, 0x2, 0xc001758500)
	test/e2e/framework/statefulset/wait.go:35 +0xbd
k8s.io/kubernetes/test/e2e/framework/statefulset.WaitForRunningAndReady(...)
	test/e2e/framework/statefulset/wait.go:80
k8s.io/kubernetes/test/e2e/apps.glob..func10.2.5()
	test/e2e/apps/statefulset.go:289 +0x326
k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func3({0x7fbd978, 0xc0007da7e0})
	vendor/github.com/onsi/ginkgo/v2/internal/node.go:449 +0x1b
k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func2()
	vendor/github.com/onsi/ginkgo/v2/internal/suite.go:750 +0x98
created by k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
	vendor/github.com/onsi/ginkgo/v2/internal/suite.go:738 +0xe3d
[AfterEach] Basic StatefulSet functionality [StatefulSetBasic]
  test/e2e/apps/statefulset.go:124
Mar 11 04:31:20.703: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/71529c8c-bfc0-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-cni-cilium-etcd.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=statefulset-6074 describe po ss-0'
Mar 11 04:31:50.780: INFO: rc: 1
Mar 11 04:31:50.780: INFO: 
Output of kubectl describe ss-0:

Mar 11 04:31:50.780: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/71529c8c-bfc0-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-cni-cilium-etcd.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=statefulset-6074 logs ss-0 --tail=100'
Mar 11 04:32:20.852: INFO: rc: 1
Mar 11 04:32:20.852: INFO: 
Last 100 log lines of ss-0:

Mar 11 04:32:20.852: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/71529c8c-bfc0-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-cni-cilium-etcd.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=statefulset-6074 describe po ss-1'
Mar 11 04:32:22.053: INFO: stderr: ""
Mar 11 04:32:22.053: INFO: stdout: "Name:             ss-1\nNamespace:        statefulset-6074\nPriority:         0\nService Account:  default\nNode:             i-0011f85bb22127167/172.20.39.97\nStart Time:       Sat, 11 Mar 2023 04:30:05 +0000\nLabels:           baz=blah\n                  controller-revision-hash=ss-b9bbc7d7b\n                  foo=bar\n                  statefulset.kubernetes.io/pod-name=ss-1\nAnnotations:      <none>\nStatus:           Running\nIP:               100.96.4.37\nIPs:\n  IP:           100.96.4.37\nControlled By:  StatefulSet/ss\nContainers:\n  webserver:\n    Container ID:   containerd://ed2e9393d50032d534af9fe6e8aa430a83944babf333990b915001d4e74cac69\n    Image:          registry.k8s.io/e2e-test-images/httpd:2.4.38-4\n    Image ID:       registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22\n    Port:           <none>\n    Host Port:      <none>\n    State:          Running\n      Started:      Sat, 11 Mar 2023 04:30:10 +0000\n    Ready:          True\n    Restart Count:  0\n    Readiness:      exec [test -f /data/statefulset-continue] delay=0s timeout=1s period=1s #success=1 #failure=1\n    Environment:    <none>\n    Mounts:\n      /data/ from datadir (rw)\n      /home from home (rw)\n      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8vw4g (ro)\nConditions:\n  Type              Status\n  Initialized       True \n  Ready             True \n  ContainersReady   True \n  PodScheduled      True \nVolumes:\n  datadir:\n    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)\n    ClaimName:  datadir-ss-1\n    ReadOnly:   false\n  home:\n    Type:          HostPath (bare host directory volume)\n    Path:          /tmp/home\n    HostPathType:  \n  kube-api-access-8vw4g:\n    Type:                    Projected (a volume that contains injected data from multiple sources)\n    TokenExpirationSeconds:  3607\n    ConfigMapName:           kube-root-ca.crt\n    ConfigMapOptional:       <nil>\n    DownwardAPI:             true\nQoS Class:                   BestEffort\nNode-Selectors:              <none>\nTolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s\n                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s\nEvents:\n  Type     Reason                  Age                    From                     Message\n  ----     ------                  ----                   ----                     -------\n  Normal   Scheduled               2m17s                  default-scheduler        Successfully assigned statefulset-6074/ss-1 to i-0011f85bb22127167\n  Normal   SuccessfulAttachVolume  2m15s                  attachdetach-controller  AttachVolume.Attach succeeded for volume \"pvc-da2666c6-5d37-4134-9dd9-3ff8b64e3690\"\n  Normal   Pulled                  2m12s                  kubelet                  Container image \"registry.k8s.io/e2e-test-images/httpd:2.4.38-4\" already present on machine\n  Normal   Created                 2m12s                  kubelet                  Created container webserver\n  Normal   Started                 2m12s                  kubelet                  Started container webserver\n  Warning  Unhealthy               113s (x22 over 2m12s)  kubelet                  Readiness probe failed:\n"
Mar 11 04:32:22.053: INFO: 
Output of kubectl describe ss-1:
Name:             ss-1
Namespace:        statefulset-6074
Priority:         0
Service Account:  default
Node:             i-0011f85bb22127167/172.20.39.97
Start Time:       Sat, 11 Mar 2023 04:30:05 +0000
Labels:           baz=blah
                  controller-revision-hash=ss-b9bbc7d7b
                  foo=bar
                  statefulset.kubernetes.io/pod-name=ss-1
Annotations:      <none>
Status:           Running
IP:               100.96.4.37
IPs:
  IP:           100.96.4.37
Controlled By:  StatefulSet/ss
Containers:
  webserver:
    Container ID:   containerd://ed2e9393d50032d534af9fe6e8aa430a83944babf333990b915001d4e74cac69
    Image:          registry.k8s.io/e2e-test-images/httpd:2.4.38-4
    Image ID:       registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sat, 11 Mar 2023 04:30:10 +0000
    Ready:          True
    Restart Count:  0
    Readiness:      exec [test -f /data/statefulset-continue] delay=0s timeout=1s period=1s #success=1 #failure=1
    Environment:    <none>
    Mounts:
      /data/ from datadir (rw)
      /home from home (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8vw4g (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  datadir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  datadir-ss-1
    ReadOnly:   false
  home:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp/home
    HostPathType:  
  kube-api-access-8vw4g:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                    From                     Message
  ----     ------                  ----                   ----                     -------
  Normal   Scheduled               2m17s                  default-scheduler        Successfully assigned statefulset-6074/ss-1 to i-0011f85bb22127167
  Normal   SuccessfulAttachVolume  2m15s                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-da2666c6-5d37-4134-9dd9-3ff8b64e3690"
  Normal   Pulled                  2m12s                  kubelet                  Container image "registry.k8s.io/e2e-test-images/httpd:2.4.38-4" already present on machine
  Normal   Created                 2m12s                  kubelet                  Created container webserver
  Normal   Started                 2m12s                  kubelet                  Started container webserver
  Warning  Unhealthy               113s (x22 over 2m12s)  kubelet                  Readiness probe failed:

Mar 11 04:32:22.053: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/71529c8c-bfc0-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-cni-cilium-etcd.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=statefulset-6074 logs ss-1 --tail=100'
Mar 11 04:32:23.025: INFO: stderr: ""
Mar 11 04:32:23.025: INFO: stdout: "[Sat Mar 11 04:30:10.459962 2023] [mpm_event:notice] [pid 1:tid 139877401697128] AH00489: Apache/2.4.38 (Unix) configured -- resuming normal operations\n[Sat Mar 11 04:30:10.460158 2023] [core:notice] [pid 1:tid 139877401697128] AH00094: Command line: 'httpd -D FOREGROUND'\n"
Mar 11 04:32:23.025: INFO: 
Last 100 log lines of ss-1:
[Sat Mar 11 04:30:10.459962 2023] [mpm_event:notice] [pid 1:tid 139877401697128] AH00489: Apache/2.4.38 (Unix) configured -- resuming normal operations
[Sat Mar 11 04:30:10.460158 2023] [core:notice] [pid 1:tid 139877401697128] AH00094: Command line: 'httpd -D FOREGROUND'

Mar 11 04:32:23.025: INFO: Deleting all statefulset in ns statefulset-6074
Mar 11 04:32:23.241: INFO: Scaling statefulset ss to 0
Mar 11 04:32:34.105: INFO: Waiting for statefulset status.replicas updated to 0
Mar 11 04:32:34.321: INFO: Deleting statefulset ss
Mar 11 04:32:34.754: INFO: Deleting pvc: datadir-ss-0 with volume pvc-d7ef4bd3-d300-49ff-84d2-c6b1f8f39237
Mar 11 04:32:34.972: INFO: Deleting pvc: datadir-ss-1 with volume pvc-da2666c6-5d37-4134-9dd9-3ff8b64e3690
Mar 11 04:32:35.404: INFO: Still waiting for pvs of statefulset to disappear:
pvc-da2666c6-5d37-4134-9dd9-3ff8b64e3690: {Phase:Released Message: Reason:}
[AfterEach] [sig-apps] StatefulSet
  test/e2e/framework/node/init/init.go:32
Mar 11 04:32:45.620: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[DeferCleanup (Each)] [sig-apps] StatefulSet
  test/e2e/framework/metrics/init/init.go:33
[DeferCleanup (Each)] [sig-apps] StatefulSet
  dump namespaces | framework.go:196
�[1mSTEP:�[0m dump namespace information after failure �[38;5;243m03/11/23 04:32:46.05�[0m
�[1mSTEP:�[0m Collecting events from namespace "statefulset-6074". �[38;5;243m03/11/23 04:32:46.05�[0m
�[1mSTEP:�[0m Found 34 events. �[38;5;243m03/11/23 04:32:46.266�[0m
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:29:44 +0000 UTC - event for datadir-ss-0: {ebs.csi.aws.com_ebs-csi-controller-85b78cc797-gcf7r_5240864c-a53d-42a7-9bf4-43eb5d699df7 } Provisioning: External provisioner is provisioning volume for claim "statefulset-6074/datadir-ss-0"
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:29:44 +0000 UTC - event for datadir-ss-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:29:44 +0000 UTC - event for datadir-ss-0: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:29:44 +0000 UTC - event for ss: {statefulset-controller } SuccessfulCreate: create Claim datadir-ss-0 Pod ss-0 in StatefulSet ss success
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:29:44 +0000 UTC - event for ss: {statefulset-controller } SuccessfulCreate: create Pod ss-0 in StatefulSet ss successful
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:29:47 +0000 UTC - event for datadir-ss-0: {ebs.csi.aws.com_ebs-csi-controller-85b78cc797-gcf7r_5240864c-a53d-42a7-9bf4-43eb5d699df7 } ProvisioningSucceeded: Successfully provisioned volume pvc-d7ef4bd3-d300-49ff-84d2-c6b1f8f39237
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:29:48 +0000 UTC - event for ss-0: {default-scheduler } Scheduled: Successfully assigned statefulset-6074/ss-0 to i-0329df98c53e1cce2
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:29:50 +0000 UTC - event for ss-0: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-d7ef4bd3-d300-49ff-84d2-c6b1f8f39237" 
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:29:51 +0000 UTC - event for ss-0: {kubelet i-0329df98c53e1cce2} Started: Started container webserver
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:29:51 +0000 UTC - event for ss-0: {kubelet i-0329df98c53e1cce2} Unhealthy: Readiness probe failed: 
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:29:51 +0000 UTC - event for ss-0: {kubelet i-0329df98c53e1cce2} Pulled: Container image "registry.k8s.io/e2e-test-images/httpd:2.4.38-4" already present on machine
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:29:51 +0000 UTC - event for ss-0: {kubelet i-0329df98c53e1cce2} Created: Created container webserver
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:01 +0000 UTC - event for datadir-ss-1: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:01 +0000 UTC - event for datadir-ss-1: {ebs.csi.aws.com_ebs-csi-controller-85b78cc797-gcf7r_5240864c-a53d-42a7-9bf4-43eb5d699df7 } Provisioning: External provisioner is provisioning volume for claim "statefulset-6074/datadir-ss-1"
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:01 +0000 UTC - event for datadir-ss-1: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:01 +0000 UTC - event for ss: {statefulset-controller } SuccessfulCreate: create Pod ss-1 in StatefulSet ss successful
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:01 +0000 UTC - event for ss: {statefulset-controller } SuccessfulCreate: create Claim datadir-ss-1 Pod ss-1 in StatefulSet ss success
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:04 +0000 UTC - event for datadir-ss-1: {ebs.csi.aws.com_ebs-csi-controller-85b78cc797-gcf7r_5240864c-a53d-42a7-9bf4-43eb5d699df7 } ProvisioningSucceeded: Successfully provisioned volume pvc-da2666c6-5d37-4134-9dd9-3ff8b64e3690
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:05 +0000 UTC - event for ss-1: {default-scheduler } Scheduled: Successfully assigned statefulset-6074/ss-1 to i-0011f85bb22127167
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:07 +0000 UTC - event for ss-1: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-da2666c6-5d37-4134-9dd9-3ff8b64e3690" 
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:10 +0000 UTC - event for ss-1: {kubelet i-0011f85bb22127167} Pulled: Container image "registry.k8s.io/e2e-test-images/httpd:2.4.38-4" already present on machine
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:10 +0000 UTC - event for ss-1: {kubelet i-0011f85bb22127167} Created: Created container webserver
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:10 +0000 UTC - event for ss-1: {kubelet i-0011f85bb22127167} Unhealthy: Readiness probe failed: 
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:10 +0000 UTC - event for ss-1: {kubelet i-0011f85bb22127167} Started: Started container webserver
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:11 +0000 UTC - event for ss-0: {default-scheduler } Scheduled: Successfully assigned statefulset-6074/ss-0 to i-0741369cdbade9399
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:11 +0000 UTC - event for ss-0: {attachdetach-controller } FailedAttachVolume: Multi-Attach error for volume "pvc-d7ef4bd3-d300-49ff-84d2-c6b1f8f39237" Volume is already exclusively attached to one node and can't be attached to another
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:11 +0000 UTC - event for ss-0: {kubelet i-0329df98c53e1cce2} Killing: Stopping container webserver
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:29 +0000 UTC - event for ss-0: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-d7ef4bd3-d300-49ff-84d2-c6b1f8f39237" 
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:30 +0000 UTC - event for ss-0: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/httpd:2.4.38-4" already present on machine
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:30 +0000 UTC - event for ss-0: {kubelet i-0741369cdbade9399} Created: Created container webserver
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:30:30 +0000 UTC - event for ss-0: {kubelet i-0741369cdbade9399} Started: Started container webserver
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:32:23 +0000 UTC - event for ss: {statefulset-controller } SuccessfulDelete: delete Pod ss-1 in StatefulSet ss successful
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:32:24 +0000 UTC - event for ss: {statefulset-controller } SuccessfulDelete: delete Pod ss-0 in StatefulSet ss successful
Mar 11 04:32:46.266: INFO: At 2023-03-11 04:32:24 +0000 UTC - event for ss-0: {kubelet i-0741369cdbade9399} Killing: Stopping container webserver
Mar 11 04:32:46.481: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Mar 11 04:32:46.481: INFO: 
Mar 11 04:32:46.697: INFO: 
Logging node info for node i-0011f85bb22127167
Mar 11 04:32:46.913: INFO: Node Info: &Node{ObjectMeta:{i-0011f85bb22127167    d4f3e0f9-f7c8-4aac-b938-1997d491f8e8 51365 0 2023-03-11 04:04:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0011f85bb22127167 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0011f85bb22127167 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0011f85bb22127167"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 04:04:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:47 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 04:04:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 04:04:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 04:04:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-03-11 04:32:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0011f85bb22127167,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 04:32:31 +0000 UTC,LastTransitionTime:2023-03-11 04:04:47 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 04:32:31 +0000 UTC,LastTransitionTime:2023-03-11 04:04:47 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 04:32:31 +0000 UTC,LastTransitionTime:2023-03-11 04:04:47 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 04:32:31 +0000 UTC,LastTransitionTime:2023-03-11 04:05:02 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.97,},NodeAddress{Type:ExternalIP,Address:54.254.3.232,},NodeAddress{Type:InternalDNS,Address:i-0011f85bb22127167.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0011f85bb22127167.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-3-232.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2cf42c34dc130918d57386d1dc0cf7,SystemUUID:ec2cf42c-34dc-1309-18d5-7386d1dc0cf7,BootID:1bccc978-bb3f-4f8e-87dd-09d834c280d3,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.26.2,KubeProxyVersion:v1.26.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.2],SizeBytes:67213512,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 04:32:46.913: INFO: 
Logging kubelet events for node i-0011f85bb22127167
Mar 11 04:32:47.135: INFO: 
Logging pods the kubelet thinks is on node i-0011f85bb22127167
Mar 11 04:32:47.354: INFO: cilium-z4lzr started at 2023-03-11 04:04:47 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:32:47.354: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 04:32:47.354: INFO: 	Container cilium-agent ready: true, restart count 3
Mar 11 04:32:47.354: INFO: coredns-559769c974-mcnnl started at 2023-03-11 04:05:20 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:32:47.354: INFO: 	Container coredns ready: true, restart count 0
Mar 11 04:32:47.354: INFO: ebs-csi-node-dr4gl started at 2023-03-11 04:04:47 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:32:47.354: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:32:47.354: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:32:47.354: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:32:48.109: INFO: 
Latency metrics for node i-0011f85bb22127167
Mar 11 04:32:48.109: INFO: 
Logging node info for node i-0329df98c53e1cce2
Mar 11 04:32:48.477: INFO: Node Info: &Node{ObjectMeta:{i-0329df98c53e1cce2    18579d8f-b168-49a3-ad57-ba94348535ca 50809 0 2023-03-11 04:04:36 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0329df98c53e1cce2 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0329df98c53e1cce2 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0329df98c53e1cce2"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-03-11 04:04:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 04:04:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:37 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 04:04:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-11 04:30:19 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0329df98c53e1cce2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:19 +0000 UTC,LastTransitionTime:2023-03-11 04:04:36 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:19 +0000 UTC,LastTransitionTime:2023-03-11 04:04:36 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:19 +0000 UTC,LastTransitionTime:2023-03-11 04:04:36 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 04:30:19 +0000 UTC,LastTransitionTime:2023-03-11 04:04:53 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.134,},NodeAddress{Type:ExternalIP,Address:54.254.230.138,},NodeAddress{Type:InternalDNS,Address:i-0329df98c53e1cce2.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0329df98c53e1cce2.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-230-138.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2707505e56c78d5055fa8514534cec,SystemUUID:ec270750-5e56-c78d-5055-fa8514534cec,BootID:e848eeb0-78cc-41f1-993e-ef001dbf3b82,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.26.2,KubeProxyVersion:v1.26.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.2],SizeBytes:67213512,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 04:32:48.477: INFO: 
Logging kubelet events for node i-0329df98c53e1cce2
Mar 11 04:32:48.696: INFO: 
Logging pods the kubelet thinks is on node i-0329df98c53e1cce2
Mar 11 04:32:48.923: INFO: ebs-csi-node-mt78m started at 2023-03-11 04:04:37 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:32:48.923: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:32:48.923: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:32:48.923: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:32:48.923: INFO: cilium-kqgwb started at 2023-03-11 04:04:37 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:32:48.923: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 04:32:48.923: INFO: 	Container cilium-agent ready: true, restart count 4
Mar 11 04:32:48.923: INFO: ebs-csi-controller-85b78cc797-gcf7r started at 2023-03-11 04:04:54 +0000 UTC (0+5 container statuses recorded)
Mar 11 04:32:48.923: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 04:32:48.923: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 04:32:48.923: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 04:32:48.923: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:32:48.923: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:32:49.678: INFO: 
Latency metrics for node i-0329df98c53e1cce2
Mar 11 04:32:49.678: INFO: 
Logging node info for node i-0741369cdbade9399
Mar 11 04:32:49.893: INFO: Node Info: &Node{ObjectMeta:{i-0741369cdbade9399    bf0e7ac5-1ef1-4825-a6a5-331297cd61eb 51342 0 2023-03-11 04:04:40 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0741369cdbade9399 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0741369cdbade9399 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0741369cdbade9399"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 04:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:40 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 04:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 04:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 04:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-03-11 04:32:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0741369cdbade9399,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 04:32:25 +0000 UTC,LastTransitionTime:2023-03-11 04:04:40 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 04:32:25 +0000 UTC,LastTransitionTime:2023-03-11 04:04:40 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 04:32:25 +0000 UTC,LastTransitionTime:2023-03-11 04:04:40 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 04:32:25 +0000 UTC,LastTransitionTime:2023-03-11 04:04:56 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.36.32,},NodeAddress{Type:ExternalIP,Address:52.221.224.194,},NodeAddress{Type:InternalDNS,Address:i-0741369cdbade9399.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0741369cdbade9399.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-224-194.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2227001f7fcddc5c884cabbabf0e0b,SystemUUID:ec222700-1f7f-cddc-5c88-4cabbabf0e0b,BootID:e32683e2-f725-4230-a02d-861048b9a290,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.26.2,KubeProxyVersion:v1.26.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.2],SizeBytes:67213512,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 04:32:49.893: INFO: 
Logging kubelet events for node i-0741369cdbade9399
Mar 11 04:32:50.112: INFO: 
Logging pods the kubelet thinks is on node i-0741369cdbade9399
Mar 11 04:32:50.336: INFO: cilium-gvqjv started at 2023-03-11 04:04:41 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:32:50.336: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 04:32:50.336: INFO: 	Container cilium-agent ready: true, restart count 3
Mar 11 04:32:50.336: INFO: funny-ip-bmnbw started at 2023-03-11 04:28:26 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:32:50.336: INFO: 	Container funny-ip ready: true, restart count 0
Mar 11 04:32:50.336: INFO: ebs-csi-node-9z96v started at 2023-03-11 04:04:41 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:32:50.336: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:32:50.336: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:32:50.336: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:32:51.070: INFO: 
Latency metrics for node i-0741369cdbade9399
Mar 11 04:32:51.070: INFO: 
Logging node info for node i-08076ffd11575a948
Mar 11 04:32:51.286: INFO: Node Info: &Node{ObjectMeta:{i-08076ffd11575a948    eb67a4db-a950-406a-a15b-0d0249beb2f5 50949 0 2023-03-11 04:02:21 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-08076ffd11575a948 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-08076ffd11575a948"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 04:02:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 04:02:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:03:06 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:03:06 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 04:03:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 04:30:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-08076ffd11575a948,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:45 +0000 UTC,LastTransitionTime:2023-03-11 04:02:19 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:45 +0000 UTC,LastTransitionTime:2023-03-11 04:02:19 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 04:30:45 +0000 UTC,LastTransitionTime:2023-03-11 04:02:19 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 04:30:45 +0000 UTC,LastTransitionTime:2023-03-11 04:03:22 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.32.13,},NodeAddress{Type:ExternalIP,Address:54.255.194.65,},NodeAddress{Type:InternalDNS,Address:i-08076ffd11575a948.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-08076ffd11575a948.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-255-194-65.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec28e5c7b8aa64aa5652371517e966ea,SystemUUID:ec28e5c7-b8aa-64aa-5652-371517e966ea,BootID:bcab411b-a8a1-415e-8bc4-977b594974ab,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.26.2,KubeProxyVersion:v1.26.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.26.2],SizeBytes:135195085,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.26.2],SizeBytes:124561719,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.2],SizeBytes:67213512,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.26.2],SizeBytes:57669942,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45226387,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43628953,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:26802430,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074660,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 04:32:51.286: INFO: 
Logging kubelet events for node i-08076ffd11575a948
Mar 11 04:32:51.505: INFO: 
Logging pods the kubelet thinks is on node i-08076ffd11575a948
Mar 11 04:32:51.947: INFO: dns-controller-bdb55cccc-wrj9v started at 2023-03-11 04:03:00 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:32:51.947: INFO: 	Container dns-controller ready: true, restart count 0
Mar 11 04:32:51.947: INFO: cilium-operator-69bd9b65dc-2bdrx started at 2023-03-11 04:03:00 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:32:51.947: INFO: 	Container cilium-operator ready: true, restart count 1
Mar 11 04:32:51.947: INFO: etcd-manager-events-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (11+1 container statuses recorded)
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 11 04:32:51.947: INFO: etcd-manager-main-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (11+1 container statuses recorded)
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 11 04:32:51.947: INFO: ebs-csi-node-wtm58 started at 2023-03-11 04:03:00 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:32:51.947: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:32:51.947: INFO: cilium-d2jhz started at 2023-03-11 04:03:00 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:32:51.947: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 11 04:32:51.947: INFO: kops-controller-msq8z started at 2023-03-11 04:03:00 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:32:51.947: INFO: 	Container kops-controller ready: true, restart count 0
Mar 11 04:32:51.947: INFO: aws-cloud-controller-manager-7spkx started at 2023-03-11 04:03:00 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:32:51.947: INFO: 	Container aws-cloud-controller-manager ready: true, restart count 0
Mar 11 04:32:51.947: INFO: etcd-manager-cilium-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (11+1 container statuses recorded)
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 11 04:32:51.947: INFO: kube-apiserver-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (0+2 container statuses recorded)
Mar 11 04:32:51.947: INFO: 	Container healthcheck ready: true, restart count 0
Mar 11 04:32:51.947: INFO: 	Container kube-apiserver ready: true, restart count 3
Mar 11 04:32:51.947: INFO: kube-controller-manager-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:32:51.947: INFO: 	Container kube-controller-manager ready: true, restart count 4
Mar 11 04:32:51.947: INFO: kube-scheduler-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:32:51.947: INFO: 	Container kube-scheduler ready: true, restart count 0
Mar 11 04:32:52.648: INFO: 
Latency metrics for node i-08076ffd11575a948
Mar 11 04:32:52.648: INFO: 
Logging node info for node i-0ceb4774ac2f61618
Mar 11 04:32:52.863: INFO: Node Info: &Node{ObjectMeta:{i-0ceb4774ac2f61618    e54bc4e4-c393-4ff9-8459-fae12254c541 50734 0 2023-03-11 04:04:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0ceb4774ac2f61618 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0ceb4774ac2f61618 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0ceb4774ac2f61618"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 04:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:36 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:36 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 04:04:36 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 04:04:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 04:30:08 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0ceb4774ac2f61618,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 04:29:58 +0000 UTC,LastTransitionTime:2023-03-11 04:04:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 04:29:58 +0000 UTC,LastTransitionTime:2023-03-11 04:04:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 04:29:58 +0000 UTC,LastTransitionTime:2023-03-11 04:04:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 04:29:58 +0000 UTC,LastTransitionTime:2023-03-11 04:04:51 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.15,},NodeAddress{Type:ExternalIP,Address:13.229.94.195,},NodeAddress{Type:InternalDNS,Address:i-0ceb4774ac2f61618.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ceb4774ac2f61618.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-229-94-195.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2b5f3d9b64f14ea7e43e834dad161f,SystemUUID:ec2b5f3d-9b64-f14e-a7e4-3e834dad161f,BootID:a024bb6c-a740-4539-b41b-e60585607279,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.26.2,KubeProxyVersion:v1.26.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.2],SizeBytes:67213512,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 04:32:52.864: INFO: 
Logging kubelet events for node i-0ceb4774ac2f61618
Mar 11 04:32:53.082: INFO: 
Logging pods the kubelet thinks is on node i-0ceb4774ac2f61618
Mar 11 04:32:53.307: INFO: cilium-9vxfg started at 2023-03-11 04:04:36 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:32:53.307: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 04:32:53.307: INFO: 	Container cilium-agent ready: true, restart count 3
Mar 11 04:32:53.307: INFO: execpodflzqr started at 2023-03-11 04:28:35 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:32:53.307: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 04:32:53.307: INFO: ebs-csi-controller-85b78cc797-xlv4h started at 2023-03-11 04:04:51 +0000 UTC (0+5 container statuses recorded)
Mar 11 04:32:53.307: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 04:32:53.307: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 04:32:53.307: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 04:32:53.307: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:32:53.307: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:32:53.307: INFO: coredns-559769c974-xdqkc started at 2023-03-11 04:04:51 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:32:53.307: INFO: 	Container coredns ready: true, restart count 0
Mar 11 04:32:53.307: INFO: coredns-autoscaler-7cb5c5b969-wrq84 started at 2023-03-11 04:04:51 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:32:53.307: INFO: 	Container autoscaler ready: true, restart count 0
Mar 11 04:32:53.307: INFO: ebs-csi-node-l4jm2 started at 2023-03-11 04:04:36 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:32:53.307: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:32:53.307: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:32:53.307: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:32:54.050: INFO: 
Latency metrics for node i-0ceb4774ac2f61618
[DeferCleanup (Each)] [sig-apps] StatefulSet
  tear down framework | framework.go:193
�[1mSTEP:�[0m Destroying namespace "statefulset-6074" for this suite. �[38;5;243m03/11/23 04:32:54.05�[0m

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

Kubernetes e2e suite [It] [sig-node] Pods Extended Pod Container Status should never report success for a pending container 1m34s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-node\]\sPods\sExtended\sPod\sContainer\sStatus\sshould\snever\sreport\ssuccess\sfor\sa\spending\scontainer$'

test/e2e/node/pods.go:474
k8s.io/kubernetes/test/e2e/node.createAndTestPodRepeatedly(0x3, 0xf, {0x7fe50a8?, 0xc005534ce0}, {0x8005640?, 0xc005534ba0})
	test/e2e/node/pods.go:474 +0x81d
k8s.io/kubernetes/test/e2e/node.glob..func10.3.2()
	test/e2e/node/pods.go:210 +0xae
				[BeforeEach] [sig-node] Pods Extended
  set up framework | framework.go:178
�[1mSTEP:�[0m Creating a kubernetes client �[38;5;243m03/11/23 04:25:22.682�[0m
Mar 11 04:25:22.682: INFO: >>> kubeConfig: /root/.kube/config
�[1mSTEP:�[0m Building a namespace api object, basename pods �[38;5;243m03/11/23 04:25:22.683�[0m
�[1mSTEP:�[0m Waiting for a default service account to be provisioned in namespace �[38;5;243m03/11/23 04:25:23.329�[0m
�[1mSTEP:�[0m Waiting for kube-root-ca.crt to be provisioned in namespace �[38;5;243m03/11/23 04:25:23.759�[0m
[BeforeEach] [sig-node] Pods Extended
  test/e2e/framework/metrics/init/init.go:31
[BeforeEach] Pod Container Status
  test/e2e/node/pods.go:204
[It] should never report success for a pending container
  test/e2e/node/pods.go:208
�[1mSTEP:�[0m creating pods that should always exit 1 and terminating the pod after a random delay �[38;5;243m03/11/23 04:25:24.187�[0m
Mar 11 04:25:27.613: INFO: watch last event seen for pod-terminate-status-1-0
Mar 11 04:25:27.613: INFO: Pod pod-terminate-status-1-0 on node i-0011f85bb22127167 t=1.545s total=3.425637351s run=2s execute=0s
Mar 11 04:25:28.413: INFO: watch last event seen for pod-terminate-status-2-0
Mar 11 04:25:28.413: INFO: Pod pod-terminate-status-2-0 on node i-0011f85bb22127167 t=16ms total=4.225041332s run=2s execute=0s
Mar 11 04:25:31.015: INFO: watch last event seen for pod-terminate-status-0-0
Mar 11 04:25:31.015: INFO: Pod pod-terminate-status-0-0 on node i-0011f85bb22127167 t=1.516s total=6.827631377s run=2s execute=0s
Mar 11 04:25:32.011: INFO: watch last event seen for pod-terminate-status-1-1
Mar 11 04:25:32.011: INFO: Pod pod-terminate-status-1-1 on node i-0011f85bb22127167 t=1.622s total=4.397533098s run=0s execute=0s
Mar 11 04:25:33.013: INFO: watch last event seen for pod-terminate-status-2-1
Mar 11 04:25:33.013: INFO: Pod pod-terminate-status-2-1 on node i-0011f85bb22127167 t=1.607s total=4.599974261s run=0s execute=0s
Mar 11 04:25:35.380: INFO: watch last event seen for pod-terminate-status-0-1
Mar 11 04:25:35.380: INFO: Pod pod-terminate-status-0-1 on node i-0741369cdbade9399 t=838ms total=4.36454319s run=0s execute=0s
Mar 11 04:25:38.977: INFO: watch last event seen for pod-terminate-status-1-2
Mar 11 04:25:38.977: INFO: Pod pod-terminate-status-1-2 on node i-0741369cdbade9399 t=1.585s total=6.965503359s run=1s execute=0s
Mar 11 04:25:40.178: INFO: watch last event seen for pod-terminate-status-2-2
Mar 11 04:25:40.178: INFO: Pod pod-terminate-status-2-2 on node i-0741369cdbade9399 t=174ms total=7.165243737s run=1s execute=0s
Mar 11 04:25:41.179: INFO: watch last event seen for pod-terminate-status-0-2
Mar 11 04:25:41.179: INFO: Pod pod-terminate-status-0-2 on node i-0741369cdbade9399 t=1.086s total=5.799168093s run=0s execute=0s
Mar 11 04:25:43.667: INFO: watch last event seen for pod-terminate-status-2-3
Mar 11 04:25:43.667: INFO: Pod pod-terminate-status-2-3 on node i-0329df98c53e1cce2 t=268ms total=3.488861146s run=1s execute=0s
Mar 11 04:25:43.897: INFO: watch last event seen for pod-terminate-status-0-3
Mar 11 04:25:43.897: INFO: Pod pod-terminate-status-0-3 on node i-0011f85bb22127167 t=597ms total=2.71755456s run=1s execute=0s
Mar 11 04:25:44.065: INFO: watch last event seen for pod-terminate-status-1-3
Mar 11 04:25:44.066: INFO: Pod pod-terminate-status-1-3 on node i-0329df98c53e1cce2 t=474ms total=5.088877262s run=1s execute=0s
Mar 11 04:25:48.578: INFO: watch last event seen for pod-terminate-status-0-4
Mar 11 04:25:48.578: INFO: Pod pod-terminate-status-0-4 on node i-0741369cdbade9399 t=1.594s total=4.681291162s run=1s execute=0s
Mar 11 04:25:49.376: INFO: watch last event seen for pod-terminate-status-2-4
Mar 11 04:25:49.377: INFO: Pod pod-terminate-status-2-4 on node i-0741369cdbade9399 t=921ms total=5.709443824s run=2s execute=0s
Mar 11 04:25:51.378: INFO: watch last event seen for pod-terminate-status-1-4
Mar 11 04:25:51.378: INFO: Pod pod-terminate-status-1-4 on node i-0741369cdbade9399 t=215ms total=7.312396783s run=1s execute=0s
Mar 11 04:25:53.069: INFO: watch last event seen for pod-terminate-status-2-5
Mar 11 04:25:53.069: INFO: Pod pod-terminate-status-2-5 on node i-0329df98c53e1cce2 t=423ms total=3.692770935s run=1s execute=0s
Mar 11 04:25:55.777: INFO: watch last event seen for pod-terminate-status-0-5
Mar 11 04:25:55.777: INFO: Pod pod-terminate-status-0-5 on node i-0741369cdbade9399 t=1.045s total=7.198907991s run=2s execute=0s
Mar 11 04:25:56.733: INFO: watch last event seen for pod-terminate-status-2-6
Mar 11 04:25:56.733: INFO: Pod pod-terminate-status-2-6 on node i-0ceb4774ac2f61618 t=1.325s total=3.663345656s run=0s execute=0s
Mar 11 04:25:56.928: INFO: watch last event seen for pod-terminate-status-1-5
Mar 11 04:25:56.928: INFO: Pod pod-terminate-status-1-5 on node i-0011f85bb22127167 t=1.822s total=5.550083401s run=1s execute=0s
Mar 11 04:25:59.329: INFO: watch last event seen for pod-terminate-status-0-6
Mar 11 04:25:59.329: INFO: Pod pod-terminate-status-0-6 on node i-0011f85bb22127167 t=220ms total=3.551839892s run=2s execute=0s
Mar 11 04:26:00.783: INFO: watch last event seen for pod-terminate-status-2-7
Mar 11 04:26:00.783: INFO: Pod pod-terminate-status-2-7 on node i-0741369cdbade9399 t=1.865s total=4.049983692s run=0s execute=0s
Mar 11 04:26:02.380: INFO: watch last event seen for pod-terminate-status-1-6
Mar 11 04:26:02.380: INFO: Pod pod-terminate-status-1-6 on node i-0741369cdbade9399 t=1.805s total=5.451629283s run=1s execute=0s
Mar 11 04:26:03.579: INFO: watch last event seen for pod-terminate-status-2-8
Mar 11 04:26:03.579: INFO: Pod pod-terminate-status-2-8 on node i-0741369cdbade9399 t=817ms total=2.796509167s run=0s execute=0s
Mar 11 04:26:05.384: INFO: watch last event seen for pod-terminate-status-0-7
Mar 11 04:26:05.384: INFO: Pod pod-terminate-status-0-7 on node i-0741369cdbade9399 t=796ms total=6.054981101s run=1s execute=0s
Mar 11 04:26:08.179: INFO: watch last event seen for pod-terminate-status-1-7
Mar 11 04:26:08.179: INFO: Pod pod-terminate-status-1-7 on node i-0741369cdbade9399 t=1.759s total=5.799232785s run=0s execute=0s
Mar 11 04:26:10.400: INFO: watch last event seen for pod-terminate-status-2-9
Mar 11 04:26:10.400: INFO: Pod pod-terminate-status-2-9 on node i-0741369cdbade9399 t=569ms total=6.820512628s run=0s execute=0s
Mar 11 04:26:11.303: INFO: watch last event seen for pod-terminate-status-1-8
Mar 11 04:26:11.303: INFO: Pod pod-terminate-status-1-8 on node i-0ceb4774ac2f61618 t=322ms total=3.123979313s run=1s execute=0s
Mar 11 04:26:17.580: INFO: watch last event seen for pod-terminate-status-0-8
Mar 11 04:26:17.580: INFO: Pod pod-terminate-status-0-8 on node i-0741369cdbade9399 t=641ms total=12.19625547s run=1s execute=0s
Mar 11 04:26:18.779: INFO: watch last event seen for pod-terminate-status-1-9
Mar 11 04:26:18.779: INFO: Pod pod-terminate-status-1-9 on node i-0741369cdbade9399 t=1.328s total=7.476054518s run=0s execute=0s
Mar 11 04:26:23.178: INFO: watch last event seen for pod-terminate-status-0-9
Mar 11 04:26:23.178: INFO: Pod pod-terminate-status-0-9 on node i-0741369cdbade9399 t=1.131s total=5.597430729s run=1s execute=0s
Mar 11 04:26:25.578: INFO: watch last event seen for pod-terminate-status-2-10
Mar 11 04:26:25.578: INFO: Pod pod-terminate-status-2-10 on node i-0741369cdbade9399 t=1.953s total=15.177721061s run=2s execute=0s
Mar 11 04:26:26.036: INFO: watch last event seen for pod-terminate-status-0-10
Mar 11 04:26:26.036: INFO: Pod pod-terminate-status-0-10 on node i-0ceb4774ac2f61618 t=968ms total=2.858556878s run=1s execute=0s
Mar 11 04:26:27.978: INFO: watch last event seen for pod-terminate-status-1-10
Mar 11 04:26:27.978: INFO: Pod pod-terminate-status-1-10 on node i-0741369cdbade9399 t=1.661s total=9.198999082s run=1s execute=0s
Mar 11 04:26:28.068: INFO: watch last event seen for pod-terminate-status-2-11
Mar 11 04:26:28.068: INFO: Pod pod-terminate-status-2-11 on node i-0ceb4774ac2f61618 t=314ms total=2.49029766s run=1s execute=0s
Mar 11 04:26:30.411: INFO: watch last event seen for pod-terminate-status-0-11
Mar 11 04:26:30.411: INFO: Pod pod-terminate-status-0-11 on node i-0ceb4774ac2f61618 t=937ms total=4.37469913s run=0s execute=0s
Mar 11 04:26:31.610: INFO: watch last event seen for pod-terminate-status-1-11
Mar 11 04:26:31.610: INFO: Pod pod-terminate-status-1-11 on node i-0ceb4774ac2f61618 t=597ms total=3.631932111s run=1s execute=0s
Mar 11 04:26:34.213: INFO: watch last event seen for pod-terminate-status-0-12
Mar 11 04:26:34.213: INFO: Pod pod-terminate-status-0-12 on node i-0ceb4774ac2f61618 t=835ms total=3.80197364s run=1s execute=0s
Mar 11 04:26:35.010: INFO: watch last event seen for pod-terminate-status-2-12
Mar 11 04:26:35.010: INFO: Pod pod-terminate-status-2-12 on node i-0ceb4774ac2f61618 t=1.532s total=6.941807691s run=1s execute=0s
Mar 11 04:26:36.022: INFO: watch last event seen for pod-terminate-status-1-12
Mar 11 04:26:36.022: INFO: Pod pod-terminate-status-1-12 on node i-0ceb4774ac2f61618 t=389ms total=4.411455974s run=1s execute=0s
Mar 11 04:26:39.613: INFO: watch last event seen for pod-terminate-status-2-13
Mar 11 04:26:39.613: INFO: Pod pod-terminate-status-2-13 on node i-0ceb4774ac2f61618 t=179ms total=4.603098842s run=1s execute=0s
Mar 11 04:26:39.799: INFO: watch last event seen for pod-terminate-status-1-13
Mar 11 04:26:39.799: INFO: Pod pod-terminate-status-1-13 on node i-0011f85bb22127167 t=1.214s total=3.776936106s run=0s execute=0s
Mar 11 04:26:40.812: INFO: watch last event seen for pod-terminate-status-0-13
Mar 11 04:26:40.812: INFO: Pod pod-terminate-status-0-13 on node i-0ceb4774ac2f61618 t=1.7s total=6.598614745s run=1s execute=0s
Mar 11 04:26:45.054: INFO: watch last event seen for pod-terminate-status-1-14
Mar 11 04:26:45.054: INFO: Pod pod-terminate-status-1-14 on node i-0741369cdbade9399 t=401ms total=5.254730694s run=1s execute=0s
Mar 11 04:26:46.056: INFO: watch last event seen for pod-terminate-status-2-14
Mar 11 04:26:46.056: INFO: Pod pod-terminate-status-2-14 on node i-0741369cdbade9399 t=1.706s total=6.442886203s run=1s execute=0s
Mar 11 04:26:47.803: INFO: watch last event seen for pod-terminate-status-0-14
Mar 11 04:26:47.803: INFO: Pod pod-terminate-status-0-14 on node i-0011f85bb22127167 t=182ms total=6.991611518s run=2s execute=0s
Mar 11 04:26:47.804: FAIL: 1 errors:
pod pod-terminate-status-0-1 on node i-0741369cdbade9399 was not Pending but has no running containers: phase=Running

Full Stack Trace
k8s.io/kubernetes/test/e2e/node.createAndTestPodRepeatedly(0x3, 0xf, {0x7fe50a8?, 0xc005534ce0}, {0x8005640?, 0xc005534ba0})
	test/e2e/node/pods.go:474 +0x81d
k8s.io/kubernetes/test/e2e/node.glob..func10.3.2()
	test/e2e/node/pods.go:210 +0xae
[AfterEach] [sig-node] Pods Extended
  test/e2e/framework/node/init/init.go:32
Mar 11 04:26:47.804: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[DeferCleanup (Each)] [sig-node] Pods Extended
  test/e2e/framework/metrics/init/init.go:33
[DeferCleanup (Each)] [sig-node] Pods Extended
  dump namespaces | framework.go:196
�[1mSTEP:�[0m dump namespace information after failure �[38;5;243m03/11/23 04:26:48.023�[0m
�[1mSTEP:�[0m Collecting events from namespace "pods-2392". �[38;5;243m03/11/23 04:26:48.023�[0m
�[1mSTEP:�[0m Found 181 events. �[38;5;243m03/11/23 04:26:48.244�[0m
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:24 +0000 UTC - event for pod-terminate-status-0-0: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-0-0 to i-0011f85bb22127167
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:24 +0000 UTC - event for pod-terminate-status-1-0: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-1-0 to i-0011f85bb22127167
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:24 +0000 UTC - event for pod-terminate-status-2-0: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-2-0 to i-0011f85bb22127167
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:26 +0000 UTC - event for pod-terminate-status-0-0: {kubelet i-0011f85bb22127167} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:26 +0000 UTC - event for pod-terminate-status-0-0: {kubelet i-0011f85bb22127167} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:26 +0000 UTC - event for pod-terminate-status-0-0: {kubelet i-0011f85bb22127167} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:26 +0000 UTC - event for pod-terminate-status-1-0: {kubelet i-0011f85bb22127167} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:26 +0000 UTC - event for pod-terminate-status-1-0: {kubelet i-0011f85bb22127167} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:26 +0000 UTC - event for pod-terminate-status-1-0: {kubelet i-0011f85bb22127167} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:26 +0000 UTC - event for pod-terminate-status-2-0: {kubelet i-0011f85bb22127167} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:26 +0000 UTC - event for pod-terminate-status-2-0: {kubelet i-0011f85bb22127167} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:26 +0000 UTC - event for pod-terminate-status-2-0: {kubelet i-0011f85bb22127167} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:27 +0000 UTC - event for pod-terminate-status-1-1: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-1-1 to i-0011f85bb22127167
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:28 +0000 UTC - event for pod-terminate-status-2-1: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-2-1 to i-0011f85bb22127167
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:29 +0000 UTC - event for pod-terminate-status-1-1: {kubelet i-0011f85bb22127167} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:29 +0000 UTC - event for pod-terminate-status-1-1: {kubelet i-0011f85bb22127167} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:29 +0000 UTC - event for pod-terminate-status-1-1: {kubelet i-0011f85bb22127167} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:30 +0000 UTC - event for pod-terminate-status-2-1: {kubelet i-0011f85bb22127167} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:30 +0000 UTC - event for pod-terminate-status-2-1: {kubelet i-0011f85bb22127167} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:30 +0000 UTC - event for pod-terminate-status-2-1: {kubelet i-0011f85bb22127167} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:31 +0000 UTC - event for pod-terminate-status-0-1: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:31 +0000 UTC - event for pod-terminate-status-0-1: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:31 +0000 UTC - event for pod-terminate-status-0-1: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-0-1 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:31 +0000 UTC - event for pod-terminate-status-0-1: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:32 +0000 UTC - event for pod-terminate-status-1-2: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-1-2 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:33 +0000 UTC - event for pod-terminate-status-1-2: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:33 +0000 UTC - event for pod-terminate-status-1-2: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:33 +0000 UTC - event for pod-terminate-status-1-2: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:33 +0000 UTC - event for pod-terminate-status-2-2: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-2-2 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:34 +0000 UTC - event for pod-terminate-status-2-2: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:34 +0000 UTC - event for pod-terminate-status-2-2: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:34 +0000 UTC - event for pod-terminate-status-2-2: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:35 +0000 UTC - event for pod-terminate-status-0-2: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-0-2 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:36 +0000 UTC - event for pod-terminate-status-0-2: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:36 +0000 UTC - event for pod-terminate-status-0-2: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:36 +0000 UTC - event for pod-terminate-status-0-2: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:39 +0000 UTC - event for pod-terminate-status-1-3: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-1-3 to i-0329df98c53e1cce2
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:39 +0000 UTC - event for pod-terminate-status-1-3: {kubelet i-0329df98c53e1cce2} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:39 +0000 UTC - event for pod-terminate-status-1-3: {kubelet i-0329df98c53e1cce2} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:40 +0000 UTC - event for pod-terminate-status-1-3: {kubelet i-0329df98c53e1cce2} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:40 +0000 UTC - event for pod-terminate-status-2-3: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-2-3 to i-0329df98c53e1cce2
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:41 +0000 UTC - event for pod-terminate-status-0-3: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-0-3 to i-0011f85bb22127167
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:41 +0000 UTC - event for pod-terminate-status-2-3: {kubelet i-0329df98c53e1cce2} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:41 +0000 UTC - event for pod-terminate-status-2-3: {kubelet i-0329df98c53e1cce2} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:41 +0000 UTC - event for pod-terminate-status-2-3: {kubelet i-0329df98c53e1cce2} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:42 +0000 UTC - event for pod-terminate-status-0-3: {kubelet i-0011f85bb22127167} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:42 +0000 UTC - event for pod-terminate-status-0-3: {kubelet i-0011f85bb22127167} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:42 +0000 UTC - event for pod-terminate-status-0-3: {kubelet i-0011f85bb22127167} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:43 +0000 UTC - event for pod-terminate-status-2-4: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-2-4 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:44 +0000 UTC - event for pod-terminate-status-0-4: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-0-4 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:44 +0000 UTC - event for pod-terminate-status-1-4: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-1-4 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:45 +0000 UTC - event for pod-terminate-status-0-4: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:45 +0000 UTC - event for pod-terminate-status-0-4: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:45 +0000 UTC - event for pod-terminate-status-0-4: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:45 +0000 UTC - event for pod-terminate-status-1-4: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:45 +0000 UTC - event for pod-terminate-status-1-4: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:45 +0000 UTC - event for pod-terminate-status-1-4: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:45 +0000 UTC - event for pod-terminate-status-2-4: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:45 +0000 UTC - event for pod-terminate-status-2-4: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:45 +0000 UTC - event for pod-terminate-status-2-4: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:48 +0000 UTC - event for pod-terminate-status-0-5: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-0-5 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:49 +0000 UTC - event for pod-terminate-status-0-5: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:49 +0000 UTC - event for pod-terminate-status-0-5: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:49 +0000 UTC - event for pod-terminate-status-2-5: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-2-5 to i-0329df98c53e1cce2
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:50 +0000 UTC - event for pod-terminate-status-0-5: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:50 +0000 UTC - event for pod-terminate-status-2-5: {kubelet i-0329df98c53e1cce2} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:50 +0000 UTC - event for pod-terminate-status-2-5: {kubelet i-0329df98c53e1cce2} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:50 +0000 UTC - event for pod-terminate-status-2-5: {kubelet i-0329df98c53e1cce2} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:51 +0000 UTC - event for pod-terminate-status-1-5: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-1-5 to i-0011f85bb22127167
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:52 +0000 UTC - event for pod-terminate-status-1-5: {kubelet i-0011f85bb22127167} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:52 +0000 UTC - event for pod-terminate-status-1-5: {kubelet i-0011f85bb22127167} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:52 +0000 UTC - event for pod-terminate-status-1-5: {kubelet i-0011f85bb22127167} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:53 +0000 UTC - event for pod-terminate-status-2-5: {kubelet i-0329df98c53e1cce2} Killing: Stopping container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:53 +0000 UTC - event for pod-terminate-status-2-6: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-2-6 to i-0ceb4774ac2f61618
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:53 +0000 UTC - event for pod-terminate-status-2-6: {kubelet i-0ceb4774ac2f61618} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:53 +0000 UTC - event for pod-terminate-status-2-6: {kubelet i-0ceb4774ac2f61618} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:53 +0000 UTC - event for pod-terminate-status-2-6: {kubelet i-0ceb4774ac2f61618} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:55 +0000 UTC - event for pod-terminate-status-0-6: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-0-6 to i-0011f85bb22127167
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:56 +0000 UTC - event for pod-terminate-status-2-7: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-2-7 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:57 +0000 UTC - event for pod-terminate-status-0-6: {kubelet i-0011f85bb22127167} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:57 +0000 UTC - event for pod-terminate-status-0-6: {kubelet i-0011f85bb22127167} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:57 +0000 UTC - event for pod-terminate-status-0-6: {kubelet i-0011f85bb22127167} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:57 +0000 UTC - event for pod-terminate-status-1-6: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-1-6 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:57 +0000 UTC - event for pod-terminate-status-2-7: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:57 +0000 UTC - event for pod-terminate-status-2-7: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:57 +0000 UTC - event for pod-terminate-status-2-7: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:58 +0000 UTC - event for pod-terminate-status-1-6: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:58 +0000 UTC - event for pod-terminate-status-1-6: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:58 +0000 UTC - event for pod-terminate-status-1-6: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:25:59 +0000 UTC - event for pod-terminate-status-0-7: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-0-7 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:00 +0000 UTC - event for pod-terminate-status-0-7: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:00 +0000 UTC - event for pod-terminate-status-0-7: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:00 +0000 UTC - event for pod-terminate-status-0-7: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:00 +0000 UTC - event for pod-terminate-status-2-8: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-2-8 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:01 +0000 UTC - event for pod-terminate-status-2-8: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:01 +0000 UTC - event for pod-terminate-status-2-8: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:01 +0000 UTC - event for pod-terminate-status-2-8: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:02 +0000 UTC - event for pod-terminate-status-1-7: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-1-7 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:03 +0000 UTC - event for pod-terminate-status-1-7: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:03 +0000 UTC - event for pod-terminate-status-1-7: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:03 +0000 UTC - event for pod-terminate-status-1-7: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:03 +0000 UTC - event for pod-terminate-status-2-9: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-2-9 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:04 +0000 UTC - event for pod-terminate-status-2-9: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:05 +0000 UTC - event for pod-terminate-status-0-8: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-0-8 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:05 +0000 UTC - event for pod-terminate-status-2-9: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:05 +0000 UTC - event for pod-terminate-status-2-9: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:06 +0000 UTC - event for pod-terminate-status-0-8: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:06 +0000 UTC - event for pod-terminate-status-0-8: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:06 +0000 UTC - event for pod-terminate-status-0-8: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:08 +0000 UTC - event for pod-terminate-status-1-8: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-1-8 to i-0ceb4774ac2f61618
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:09 +0000 UTC - event for pod-terminate-status-1-8: {kubelet i-0ceb4774ac2f61618} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:09 +0000 UTC - event for pod-terminate-status-1-8: {kubelet i-0ceb4774ac2f61618} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:09 +0000 UTC - event for pod-terminate-status-1-8: {kubelet i-0ceb4774ac2f61618} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:10 +0000 UTC - event for pod-terminate-status-2-10: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-2-10 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:11 +0000 UTC - event for pod-terminate-status-1-9: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-1-9 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:12 +0000 UTC - event for pod-terminate-status-1-9: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:12 +0000 UTC - event for pod-terminate-status-1-9: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:12 +0000 UTC - event for pod-terminate-status-1-9: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:12 +0000 UTC - event for pod-terminate-status-2-10: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:12 +0000 UTC - event for pod-terminate-status-2-10: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:12 +0000 UTC - event for pod-terminate-status-2-10: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:17 +0000 UTC - event for pod-terminate-status-0-9: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-0-9 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:18 +0000 UTC - event for pod-terminate-status-0-9: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:18 +0000 UTC - event for pod-terminate-status-0-9: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:18 +0000 UTC - event for pod-terminate-status-0-9: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:18 +0000 UTC - event for pod-terminate-status-1-10: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-1-10 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:19 +0000 UTC - event for pod-terminate-status-1-10: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:19 +0000 UTC - event for pod-terminate-status-1-10: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:19 +0000 UTC - event for pod-terminate-status-1-10: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:23 +0000 UTC - event for pod-terminate-status-0-10: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-0-10 to i-0ceb4774ac2f61618
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:24 +0000 UTC - event for pod-terminate-status-0-10: {kubelet i-0ceb4774ac2f61618} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:24 +0000 UTC - event for pod-terminate-status-0-10: {kubelet i-0ceb4774ac2f61618} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:24 +0000 UTC - event for pod-terminate-status-0-10: {kubelet i-0ceb4774ac2f61618} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:25 +0000 UTC - event for pod-terminate-status-2-11: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-2-11 to i-0ceb4774ac2f61618
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:26 +0000 UTC - event for pod-terminate-status-0-11: {kubelet i-0ceb4774ac2f61618} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:26 +0000 UTC - event for pod-terminate-status-0-11: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-0-11 to i-0ceb4774ac2f61618
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:26 +0000 UTC - event for pod-terminate-status-0-11: {kubelet i-0ceb4774ac2f61618} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:26 +0000 UTC - event for pod-terminate-status-0-11: {kubelet i-0ceb4774ac2f61618} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:26 +0000 UTC - event for pod-terminate-status-2-11: {kubelet i-0ceb4774ac2f61618} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:26 +0000 UTC - event for pod-terminate-status-2-11: {kubelet i-0ceb4774ac2f61618} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:26 +0000 UTC - event for pod-terminate-status-2-11: {kubelet i-0ceb4774ac2f61618} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:28 +0000 UTC - event for pod-terminate-status-1-11: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-1-11 to i-0ceb4774ac2f61618
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:28 +0000 UTC - event for pod-terminate-status-2-12: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-2-12 to i-0ceb4774ac2f61618
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:29 +0000 UTC - event for pod-terminate-status-1-11: {kubelet i-0ceb4774ac2f61618} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:29 +0000 UTC - event for pod-terminate-status-1-11: {kubelet i-0ceb4774ac2f61618} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:29 +0000 UTC - event for pod-terminate-status-1-11: {kubelet i-0ceb4774ac2f61618} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:29 +0000 UTC - event for pod-terminate-status-2-12: {kubelet i-0ceb4774ac2f61618} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:29 +0000 UTC - event for pod-terminate-status-2-12: {kubelet i-0ceb4774ac2f61618} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:29 +0000 UTC - event for pod-terminate-status-2-12: {kubelet i-0ceb4774ac2f61618} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:30 +0000 UTC - event for pod-terminate-status-0-12: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-0-12 to i-0ceb4774ac2f61618
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:31 +0000 UTC - event for pod-terminate-status-0-12: {kubelet i-0ceb4774ac2f61618} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:31 +0000 UTC - event for pod-terminate-status-0-12: {kubelet i-0ceb4774ac2f61618} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:31 +0000 UTC - event for pod-terminate-status-0-12: {kubelet i-0ceb4774ac2f61618} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:31 +0000 UTC - event for pod-terminate-status-1-12: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-1-12 to i-0ceb4774ac2f61618
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:32 +0000 UTC - event for pod-terminate-status-1-12: {kubelet i-0ceb4774ac2f61618} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:32 +0000 UTC - event for pod-terminate-status-1-12: {kubelet i-0ceb4774ac2f61618} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:32 +0000 UTC - event for pod-terminate-status-1-12: {kubelet i-0ceb4774ac2f61618} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:34 +0000 UTC - event for pod-terminate-status-0-13: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-0-13 to i-0ceb4774ac2f61618
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:35 +0000 UTC - event for pod-terminate-status-0-13: {kubelet i-0ceb4774ac2f61618} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:35 +0000 UTC - event for pod-terminate-status-0-13: {kubelet i-0ceb4774ac2f61618} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:35 +0000 UTC - event for pod-terminate-status-0-13: {kubelet i-0ceb4774ac2f61618} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:35 +0000 UTC - event for pod-terminate-status-2-13: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-2-13 to i-0ceb4774ac2f61618
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:36 +0000 UTC - event for pod-terminate-status-1-13: {kubelet i-0011f85bb22127167} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:36 +0000 UTC - event for pod-terminate-status-1-13: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-1-13 to i-0011f85bb22127167
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:36 +0000 UTC - event for pod-terminate-status-1-13: {kubelet i-0011f85bb22127167} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:36 +0000 UTC - event for pod-terminate-status-1-13: {kubelet i-0011f85bb22127167} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:36 +0000 UTC - event for pod-terminate-status-2-13: {kubelet i-0ceb4774ac2f61618} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:36 +0000 UTC - event for pod-terminate-status-2-13: {kubelet i-0ceb4774ac2f61618} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:36 +0000 UTC - event for pod-terminate-status-2-13: {kubelet i-0ceb4774ac2f61618} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:39 +0000 UTC - event for pod-terminate-status-1-14: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-1-14 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:39 +0000 UTC - event for pod-terminate-status-2-14: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-2-14 to i-0741369cdbade9399
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:40 +0000 UTC - event for pod-terminate-status-0-14: {default-scheduler } Scheduled: Successfully assigned pods-2392/pod-terminate-status-0-14 to i-0011f85bb22127167
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:40 +0000 UTC - event for pod-terminate-status-1-14: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:40 +0000 UTC - event for pod-terminate-status-1-14: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:40 +0000 UTC - event for pod-terminate-status-1-14: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:40 +0000 UTC - event for pod-terminate-status-2-14: {kubelet i-0741369cdbade9399} Created: Created container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:40 +0000 UTC - event for pod-terminate-status-2-14: {kubelet i-0741369cdbade9399} Started: Started container fail
Mar 11 04:26:48.244: INFO: At 2023-03-11 04:26:40 +0000 UTC - event for pod-terminate-status-2-14: {kubelet i-0741369cdbade9399} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.245: INFO: At 2023-03-11 04:26:41 +0000 UTC - event for pod-terminate-status-0-14: {kubelet i-0011f85bb22127167} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine
Mar 11 04:26:48.245: INFO: At 2023-03-11 04:26:41 +0000 UTC - event for pod-terminate-status-0-14: {kubelet i-0011f85bb22127167} Created: Created container fail
Mar 11 04:26:48.245: INFO: At 2023-03-11 04:26:42 +0000 UTC - event for pod-terminate-status-0-14: {kubelet i-0011f85bb22127167} Started: Started container fail
Mar 11 04:26:48.460: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Mar 11 04:26:48.460: INFO: 
Mar 11 04:26:48.696: INFO: 
Logging node info for node i-0011f85bb22127167
Mar 11 04:26:48.913: INFO: Node Info: &Node{ObjectMeta:{i-0011f85bb22127167    d4f3e0f9-f7c8-4aac-b938-1997d491f8e8 43379 0 2023-03-11 04:04:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0011f85bb22127167 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0011f85bb22127167 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-6431":"i-0011f85bb22127167","ebs.csi.aws.com":"i-0011f85bb22127167"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 04:04:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:47 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 04:04:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 04:04:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 04:04:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-11 04:26:45 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-11 04:26:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0011f85bb22127167,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 04:26:45 +0000 UTC,LastTransitionTime:2023-03-11 04:04:47 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 04:26:45 +0000 UTC,LastTransitionTime:2023-03-11 04:04:47 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 04:26:45 +0000 UTC,LastTransitionTime:2023-03-11 04:04:47 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 04:26:45 +0000 UTC,LastTransitionTime:2023-03-11 04:05:02 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.39.97,},NodeAddress{Type:ExternalIP,Address:54.254.3.232,},NodeAddress{Type:InternalDNS,Address:i-0011f85bb22127167.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0011f85bb22127167.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-3-232.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2cf42c34dc130918d57386d1dc0cf7,SystemUUID:ec2cf42c-34dc-1309-18d5-7386d1dc0cf7,BootID:1bccc978-bb3f-4f8e-87dd-09d834c280d3,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.26.2,KubeProxyVersion:v1.26.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.2],SizeBytes:67213512,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-00deca66fd6eb3752],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-00deca66fd6eb3752,DevicePath:,},},Config:nil,},}
Mar 11 04:26:48.913: INFO: 
Logging kubelet events for node i-0011f85bb22127167
Mar 11 04:26:49.131: INFO: 
Logging pods the kubelet thinks is on node i-0011f85bb22127167
Mar 11 04:26:49.359: INFO: private started at 2023-03-11 04:24:59 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:49.359: INFO: 	Container cntr ready: true, restart count 0
Mar 11 04:26:49.359: INFO: netserver-0 started at 2023-03-11 04:26:06 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:49.359: INFO: 	Container webserver ready: true, restart count 0
Mar 11 04:26:49.359: INFO: csi-mockplugin-attacher-0 started at 2023-03-11 04:26:14 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:49.359: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 04:26:49.359: INFO: pod-subpath-test-configmap-g4p8 started at 2023-03-11 04:26:41 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:49.359: INFO: 	Container test-container-subpath-configmap-g4p8 ready: false, restart count 0
Mar 11 04:26:49.359: INFO: csi-mockplugin-0 started at 2023-03-11 04:26:13 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:26:49.359: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 04:26:49.359: INFO: 	Container driver-registrar ready: true, restart count 0
Mar 11 04:26:49.359: INFO: 	Container mock ready: true, restart count 0
Mar 11 04:26:49.359: INFO: netserver-0 started at 2023-03-11 04:26:15 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:49.359: INFO: 	Container webserver ready: true, restart count 0
Mar 11 04:26:49.359: INFO: host-test-container-pod started at 2023-03-11 04:26:40 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:49.359: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 04:26:49.359: INFO: cilium-z4lzr started at 2023-03-11 04:04:47 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:26:49.359: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 04:26:49.359: INFO: 	Container cilium-agent ready: true, restart count 3
Mar 11 04:26:49.359: INFO: pod-subpath-test-preprovisionedpv-zxnh started at 2023-03-11 04:26:42 +0000 UTC (2+1 container statuses recorded)
Mar 11 04:26:49.359: INFO: 	Init container init-volume-preprovisionedpv-zxnh ready: false, restart count 0
Mar 11 04:26:49.359: INFO: 	Init container test-init-volume-preprovisionedpv-zxnh ready: false, restart count 0
Mar 11 04:26:49.359: INFO: 	Container test-container-subpath-preprovisionedpv-zxnh ready: false, restart count 0
Mar 11 04:26:49.359: INFO: hostexec-i-0011f85bb22127167-rhp6t started at <nil> (0+0 container statuses recorded)
Mar 11 04:26:49.360: INFO: coredns-559769c974-mcnnl started at 2023-03-11 04:05:20 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:49.360: INFO: 	Container coredns ready: true, restart count 0
Mar 11 04:26:49.360: INFO: master started at 2023-03-11 04:24:53 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:49.360: INFO: 	Container cntr ready: true, restart count 0
Mar 11 04:26:49.360: INFO: hostexec-i-0011f85bb22127167-vcjr6 started at 2023-03-11 04:26:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:49.360: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 04:26:49.360: INFO: inline-volume-tester-4xlx7 started at 2023-03-11 04:25:57 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:49.360: INFO: 	Container csi-volume-tester ready: true, restart count 0
Mar 11 04:26:49.360: INFO: ebs-csi-node-dr4gl started at 2023-03-11 04:04:47 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:26:49.360: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:26:49.360: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:26:49.360: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:26:49.360: INFO: hostexec-i-0011f85bb22127167-kn728 started at 2023-03-11 04:26:42 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:49.360: INFO: 	Container agnhost-container ready: false, restart count 0
Mar 11 04:26:49.360: INFO: default started at 2023-03-11 04:25:03 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:49.360: INFO: 	Container cntr ready: true, restart count 0
Mar 11 04:26:49.360: INFO: csi-hostpathplugin-0 started at <nil> (0+0 container statuses recorded)
Mar 11 04:26:49.360: INFO: hostexec-i-0011f85bb22127167-vmxqw started at 2023-03-11 04:26:41 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:49.360: INFO: 	Container agnhost-container ready: false, restart count 0
Mar 11 04:26:49.360: INFO: slave started at 2023-03-11 04:24:56 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:49.360: INFO: 	Container cntr ready: true, restart count 0
Mar 11 04:26:50.621: INFO: 
Latency metrics for node i-0011f85bb22127167
Mar 11 04:26:50.621: INFO: 
Logging node info for node i-0329df98c53e1cce2
Mar 11 04:26:50.836: INFO: Node Info: &Node{ObjectMeta:{i-0329df98c53e1cce2    18579d8f-b168-49a3-ad57-ba94348535ca 43608 0 2023-03-11 04:04:36 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0329df98c53e1cce2 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0329df98c53e1cce2 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-8572":"i-0329df98c53e1cce2","ebs.csi.aws.com":"i-0329df98c53e1cce2"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-03-11 04:04:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 04:04:36 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:37 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 04:04:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-11 04:26:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {kube-controller-manager Update v1 2023-03-11 04:26:50 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0329df98c53e1cce2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229167104 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124309504 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 04:26:24 +0000 UTC,LastTransitionTime:2023-03-11 04:04:36 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 04:26:24 +0000 UTC,LastTransitionTime:2023-03-11 04:04:36 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 04:26:24 +0000 UTC,LastTransitionTime:2023-03-11 04:04:36 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 04:26:24 +0000 UTC,LastTransitionTime:2023-03-11 04:04:53 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.134,},NodeAddress{Type:ExternalIP,Address:54.254.230.138,},NodeAddress{Type:InternalDNS,Address:i-0329df98c53e1cce2.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0329df98c53e1cce2.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-230-138.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2707505e56c78d5055fa8514534cec,SystemUUID:ec270750-5e56-c78d-5055-fa8514534cec,BootID:e848eeb0-78cc-41f1-993e-ef001dbf3b82,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.26.2,KubeProxyVersion:v1.26.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.2],SizeBytes:67213512,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-042c53f490012b9b1,DevicePath:,},},Config:nil,},}
Mar 11 04:26:50.837: INFO: 
Logging kubelet events for node i-0329df98c53e1cce2
Mar 11 04:26:51.055: INFO: 
Logging pods the kubelet thinks is on node i-0329df98c53e1cce2
Mar 11 04:26:51.277: INFO: pod-subpath-test-inlinevolume-zscd started at 2023-03-11 04:26:43 +0000 UTC (2+1 container statuses recorded)
Mar 11 04:26:51.277: INFO: 	Init container init-volume-inlinevolume-zscd ready: true, restart count 0
Mar 11 04:26:51.277: INFO: 	Init container test-init-volume-inlinevolume-zscd ready: true, restart count 0
Mar 11 04:26:51.277: INFO: 	Container test-container-subpath-inlinevolume-zscd ready: false, restart count 0
Mar 11 04:26:51.277: INFO: adopt-release-fh95m started at 2023-03-11 04:26:48 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:51.277: INFO: 	Container c ready: false, restart count 0
Mar 11 04:26:51.277: INFO: ebs-csi-controller-85b78cc797-gcf7r started at 2023-03-11 04:04:54 +0000 UTC (0+5 container statuses recorded)
Mar 11 04:26:51.277: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 04:26:51.277: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 04:26:51.277: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 04:26:51.277: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:26:51.277: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:26:51.277: INFO: hostexec-i-0329df98c53e1cce2-mtdqd started at 2023-03-11 04:26:32 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:51.277: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 04:26:51.277: INFO: pod-subpath-test-dynamicpv-d2sr started at 2023-03-11 04:26:48 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:26:51.277: INFO: 	Init container init-volume-dynamicpv-d2sr ready: false, restart count 0
Mar 11 04:26:51.277: INFO: 	Container test-container-subpath-dynamicpv-d2sr ready: false, restart count 0
Mar 11 04:26:51.277: INFO: netserver-1 started at 2023-03-11 04:26:15 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:51.277: INFO: 	Container webserver ready: true, restart count 0
Mar 11 04:26:51.277: INFO: pvc-volume-tester-gcv8x started at 2023-03-11 04:26:34 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:51.277: INFO: 	Container volume-tester ready: true, restart count 0
Mar 11 04:26:51.277: INFO: pod-fd6ef5b5-d4a1-49fc-aeee-3e0042a23cdf started at 2023-03-11 04:26:42 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:51.277: INFO: 	Container write-pod ready: true, restart count 0
Mar 11 04:26:51.277: INFO: csi-mockplugin-0 started at 2023-03-11 04:26:21 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:26:51.277: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 04:26:51.277: INFO: 	Container driver-registrar ready: true, restart count 0
Mar 11 04:26:51.277: INFO: 	Container mock ready: true, restart count 0
Mar 11 04:26:51.277: INFO: ebs-csi-node-mt78m started at 2023-03-11 04:04:37 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:26:51.277: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:26:51.277: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:26:51.277: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:26:51.277: INFO: hostexec-i-0329df98c53e1cce2-jv4kv started at 2023-03-11 04:26:45 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:51.277: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 04:26:51.277: INFO: liveness-a2233b1f-ca8e-4ec5-8790-c497cfc68d0d started at 2023-03-11 04:23:16 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:51.277: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 04:26:51.277: INFO: cilium-kqgwb started at 2023-03-11 04:04:37 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:26:51.277: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 04:26:51.277: INFO: 	Container cilium-agent ready: true, restart count 4
Mar 11 04:26:52.470: INFO: 
Latency metrics for node i-0329df98c53e1cce2
Mar 11 04:26:52.470: INFO: 
Logging node info for node i-0741369cdbade9399
Mar 11 04:26:52.686: INFO: Node Info: &Node{ObjectMeta:{i-0741369cdbade9399    bf0e7ac5-1ef1-4825-a6a5-331297cd61eb 43335 0 2023-03-11 04:04:40 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0741369cdbade9399 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0741369cdbade9399 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1757":"i-0741369cdbade9399","csi-mock-csi-mock-volumes-2323":"i-0741369cdbade9399","ebs.csi.aws.com":"i-0741369cdbade9399"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 04:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:40 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 04:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 04:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 04:04:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-03-11 04:26:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0741369cdbade9399,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 04:26:38 +0000 UTC,LastTransitionTime:2023-03-11 04:04:40 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 04:26:38 +0000 UTC,LastTransitionTime:2023-03-11 04:04:40 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 04:26:38 +0000 UTC,LastTransitionTime:2023-03-11 04:04:40 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 04:26:38 +0000 UTC,LastTransitionTime:2023-03-11 04:04:56 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.36.32,},NodeAddress{Type:ExternalIP,Address:52.221.224.194,},NodeAddress{Type:InternalDNS,Address:i-0741369cdbade9399.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0741369cdbade9399.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-221-224-194.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2227001f7fcddc5c884cabbabf0e0b,SystemUUID:ec222700-1f7f-cddc-5c88-4cabbabf0e0b,BootID:e32683e2-f725-4230-a02d-861048b9a290,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.26.2,KubeProxyVersion:v1.26.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.2],SizeBytes:67213512,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 04:26:52.686: INFO: 
Logging kubelet events for node i-0741369cdbade9399
Mar 11 04:26:52.905: INFO: 
Logging pods the kubelet thinks is on node i-0741369cdbade9399
Mar 11 04:26:53.125: INFO: test-container-pod started at 2023-03-11 04:26:40 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:53.125: INFO: 	Container webserver ready: true, restart count 0
Mar 11 04:26:53.125: INFO: liveness-80626b50-ba28-40ac-852a-f8e18b65b758 started at 2023-03-11 04:26:22 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:53.125: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 04:26:53.125: INFO: inline-volume-ww7q7 started at 2023-03-11 04:26:49 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:53.125: INFO: 	Container volume-tester ready: false, restart count 0
Mar 11 04:26:53.125: INFO: adopt-release-xm8gk started at 2023-03-11 04:26:48 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:53.125: INFO: 	Container c ready: true, restart count 0
Mar 11 04:26:53.125: INFO: csi-hostpathplugin-0 started at 2023-03-11 04:24:41 +0000 UTC (0+7 container statuses recorded)
Mar 11 04:26:53.125: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 04:26:53.125: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 04:26:53.125: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 04:26:53.125: INFO: 	Container csi-snapshotter ready: true, restart count 0
Mar 11 04:26:53.125: INFO: 	Container hostpath ready: true, restart count 0
Mar 11 04:26:53.125: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:26:53.125: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:26:53.125: INFO: csi-mockplugin-0 started at 2023-03-11 04:26:40 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:26:53.125: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 04:26:53.125: INFO: 	Container driver-registrar ready: true, restart count 0
Mar 11 04:26:53.125: INFO: 	Container mock ready: true, restart count 0
Mar 11 04:26:53.125: INFO: csi-mockplugin-attacher-0 started at 2023-03-11 04:26:41 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:53.125: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 04:26:53.125: INFO: pvc-volume-tester-ddqd8 started at 2023-03-11 04:26:47 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:53.125: INFO: 	Container volume-tester ready: true, restart count 0
Mar 11 04:26:53.125: INFO: netserver-2 started at 2023-03-11 04:26:51 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:53.125: INFO: 	Container webserver ready: false, restart count 0
Mar 11 04:26:53.125: INFO: netserver-2 started at 2023-03-11 04:26:16 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:53.125: INFO: 	Container webserver ready: true, restart count 0
Mar 11 04:26:53.125: INFO: cilium-gvqjv started at 2023-03-11 04:04:41 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:26:53.125: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 04:26:53.125: INFO: 	Container cilium-agent ready: true, restart count 3
Mar 11 04:26:53.126: INFO: ebs-csi-node-9z96v started at 2023-03-11 04:04:41 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:26:53.126: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:26:53.126: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:26:53.126: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:26:54.125: INFO: 
Latency metrics for node i-0741369cdbade9399
Mar 11 04:26:54.125: INFO: 
Logging node info for node i-08076ffd11575a948
Mar 11 04:26:54.341: INFO: Node Info: &Node{ObjectMeta:{i-08076ffd11575a948    eb67a4db-a950-406a-a15b-0d0249beb2f5 40868 0 2023-03-11 04:02:21 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-08076ffd11575a948 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-08076ffd11575a948"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 04:02:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 04:02:50 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:03:06 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:03:06 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 04:03:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 04:25:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-08076ffd11575a948,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862904832 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3758047232 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 04:25:39 +0000 UTC,LastTransitionTime:2023-03-11 04:02:19 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 04:25:39 +0000 UTC,LastTransitionTime:2023-03-11 04:02:19 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 04:25:39 +0000 UTC,LastTransitionTime:2023-03-11 04:02:19 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 04:25:39 +0000 UTC,LastTransitionTime:2023-03-11 04:03:22 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.32.13,},NodeAddress{Type:ExternalIP,Address:54.255.194.65,},NodeAddress{Type:InternalDNS,Address:i-08076ffd11575a948.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-08076ffd11575a948.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-255-194-65.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec28e5c7b8aa64aa5652371517e966ea,SystemUUID:ec28e5c7-b8aa-64aa-5652-371517e966ea,BootID:bcab411b-a8a1-415e-8bc4-977b594974ab,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.26.2,KubeProxyVersion:v1.26.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.26.2],SizeBytes:135195085,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.26.2],SizeBytes:124561719,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.2],SizeBytes:67213512,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.26.2],SizeBytes:57669942,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45226387,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43628953,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:26802430,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074660,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 04:26:54.341: INFO: 
Logging kubelet events for node i-08076ffd11575a948
Mar 11 04:26:54.561: INFO: 
Logging pods the kubelet thinks is on node i-08076ffd11575a948
Mar 11 04:26:54.787: INFO: dns-controller-bdb55cccc-wrj9v started at 2023-03-11 04:03:00 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:54.787: INFO: 	Container dns-controller ready: true, restart count 0
Mar 11 04:26:54.787: INFO: cilium-operator-69bd9b65dc-2bdrx started at 2023-03-11 04:03:00 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:54.787: INFO: 	Container cilium-operator ready: true, restart count 1
Mar 11 04:26:54.787: INFO: etcd-manager-events-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (11+1 container statuses recorded)
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 11 04:26:54.787: INFO: etcd-manager-main-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (11+1 container statuses recorded)
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 11 04:26:54.787: INFO: ebs-csi-node-wtm58 started at 2023-03-11 04:03:00 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:26:54.787: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:26:54.787: INFO: cilium-d2jhz started at 2023-03-11 04:03:00 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:26:54.787: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 11 04:26:54.787: INFO: kops-controller-msq8z started at 2023-03-11 04:03:00 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:54.787: INFO: 	Container kops-controller ready: true, restart count 0
Mar 11 04:26:54.787: INFO: aws-cloud-controller-manager-7spkx started at 2023-03-11 04:03:00 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:54.787: INFO: 	Container aws-cloud-controller-manager ready: true, restart count 0
Mar 11 04:26:54.787: INFO: etcd-manager-cilium-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (11+1 container statuses recorded)
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 11 04:26:54.787: INFO: kube-apiserver-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (0+2 container statuses recorded)
Mar 11 04:26:54.787: INFO: 	Container healthcheck ready: true, restart count 0
Mar 11 04:26:54.787: INFO: 	Container kube-apiserver ready: true, restart count 3
Mar 11 04:26:54.787: INFO: kube-controller-manager-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:54.787: INFO: 	Container kube-controller-manager ready: true, restart count 4
Mar 11 04:26:54.787: INFO: kube-scheduler-i-08076ffd11575a948 started at 2023-03-11 04:00:39 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:54.787: INFO: 	Container kube-scheduler ready: true, restart count 0
Mar 11 04:26:55.516: INFO: 
Latency metrics for node i-08076ffd11575a948
Mar 11 04:26:55.516: INFO: 
Logging node info for node i-0ceb4774ac2f61618
Mar 11 04:26:55.732: INFO: Node Info: &Node{ObjectMeta:{i-0ceb4774ac2f61618    e54bc4e4-c393-4ff9-8459-fae12254c541 43755 0 2023-03-11 04:04:35 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0ceb4774ac2f61618 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0ceb4774ac2f61618 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-3261":"i-0ceb4774ac2f61618","csi-mock-csi-mock-volumes-203":"i-0ceb4774ac2f61618","ebs.csi.aws.com":"i-0ceb4774ac2f61618"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 04:04:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:36 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 04:04:36 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 04:04:36 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 04:04:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kube-controller-manager Update v1 2023-03-11 04:26:45 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-11 04:26:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0ceb4774ac2f61618,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8229175296 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{8124317696 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 04:26:54 +0000 UTC,LastTransitionTime:2023-03-11 04:04:35 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 04:26:54 +0000 UTC,LastTransitionTime:2023-03-11 04:04:35 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 04:26:54 +0000 UTC,LastTransitionTime:2023-03-11 04:04:35 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 04:26:54 +0000 UTC,LastTransitionTime:2023-03-11 04:04:51 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.15,},NodeAddress{Type:ExternalIP,Address:13.229.94.195,},NodeAddress{Type:InternalDNS,Address:i-0ceb4774ac2f61618.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ceb4774ac2f61618.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-229-94-195.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2b5f3d9b64f14ea7e43e834dad161f,SystemUUID:ec2b5f3d-9b64-f14e-a7e4-3e834dad161f,BootID:a024bb6c-a740-4539-b41b-e60585607279,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.26.2,KubeProxyVersion:v1.26.2,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:166719855,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.26.2],SizeBytes:67213512,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:25491225,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-3261^edb143d2-bfc4-11ed-a15a-c67c76048c90 kubernetes.io/csi/csi-mock-csi-mock-volumes-203^ae6d3f01-bfc4-11ed-9705-86a9588529cf],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-3261^edb143d2-bfc4-11ed-a15a-c67c76048c90,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-203^ae6d3f01-bfc4-11ed-9705-86a9588529cf,DevicePath:,},},Config:nil,},}
Mar 11 04:26:55.733: INFO: 
Logging kubelet events for node i-0ceb4774ac2f61618
Mar 11 04:26:55.953: INFO: 
Logging pods the kubelet thinks is on node i-0ceb4774ac2f61618
Mar 11 04:26:56.178: INFO: csi-mockplugin-0 started at 2023-03-11 04:24:51 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:26:56.178: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 04:26:56.178: INFO: 	Container driver-registrar ready: true, restart count 0
Mar 11 04:26:56.178: INFO: 	Container mock ready: true, restart count 0
Mar 11 04:26:56.178: INFO: pod-disruption-failure-ignore-2-hp9c6 started at 2023-03-11 04:26:53 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:56.178: INFO: 	Container c ready: false, restart count 0
Mar 11 04:26:56.178: INFO: ebs-csi-controller-85b78cc797-xlv4h started at 2023-03-11 04:04:51 +0000 UTC (0+5 container statuses recorded)
Mar 11 04:26:56.178: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 04:26:56.178: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 04:26:56.178: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 04:26:56.178: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:26:56.178: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:26:56.178: INFO: coredns-559769c974-xdqkc started at 2023-03-11 04:04:51 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:56.178: INFO: 	Container coredns ready: true, restart count 0
Mar 11 04:26:56.178: INFO: csi-mockplugin-attacher-0 started at 2023-03-11 04:24:51 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:56.178: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 04:26:56.178: INFO: pvc-volume-tester-8rwxw started at 2023-03-11 04:24:58 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:56.178: INFO: 	Container volume-tester ready: true, restart count 0
Mar 11 04:26:56.178: INFO: netserver-3 started at 2023-03-11 04:26:51 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:56.178: INFO: 	Container webserver ready: false, restart count 0
Mar 11 04:26:56.178: INFO: pod-disruption-failure-ignore-0-8l7pf started at 2023-03-11 04:26:40 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:56.178: INFO: 	Container c ready: true, restart count 0
Mar 11 04:26:56.178: INFO: inline-volume-tester-5tgl2 started at 2023-03-11 04:26:45 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:56.178: INFO: 	Container csi-volume-tester ready: true, restart count 0
Mar 11 04:26:56.178: INFO: coredns-autoscaler-7cb5c5b969-wrq84 started at 2023-03-11 04:04:51 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:56.178: INFO: 	Container autoscaler ready: true, restart count 0
Mar 11 04:26:56.178: INFO: csi-hostpathplugin-0 started at 2023-03-11 04:26:38 +0000 UTC (0+7 container statuses recorded)
Mar 11 04:26:56.178: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 04:26:56.178: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 04:26:56.178: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 04:26:56.178: INFO: 	Container csi-snapshotter ready: true, restart count 0
Mar 11 04:26:56.178: INFO: 	Container hostpath ready: true, restart count 0
Mar 11 04:26:56.178: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:26:56.178: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:26:56.178: INFO: ebs-csi-node-l4jm2 started at 2023-03-11 04:04:36 +0000 UTC (0+3 container statuses recorded)
Mar 11 04:26:56.178: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 04:26:56.178: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 04:26:56.178: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 04:26:56.178: INFO: cilium-9vxfg started at 2023-03-11 04:04:36 +0000 UTC (1+1 container statuses recorded)
Mar 11 04:26:56.178: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 04:26:56.178: INFO: 	Container cilium-agent ready: true, restart count 3
Mar 11 04:26:56.178: INFO: pod-disruption-failure-ignore-1-2gpg8 started at 2023-03-11 04:26:40 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:56.178: INFO: 	Container c ready: false, restart count 0
Mar 11 04:26:56.178: INFO: pod-disruption-failure-ignore-1-hpn2g started at 2023-03-11 04:26:45 +0000 UTC (0+1 container statuses recorded)
Mar 11 04:26:56.178: INFO: 	Container c ready: false, restart count 0
Mar 11 04:26:56.975: INFO: 
Latency metrics for node i-0ceb4774ac2f61618
[DeferCleanup (Each)] [sig-node] Pods Extended
  tear down framework | framework.go:193
�[1mSTEP:�[0m Destroying namespace "pods-2392" for this suite. �[38;5;243m03/11/23 04:26:56.975�[0m

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

kubetest2 Test 27m3s

exit status 255
				from junit_runner.xml

Filter through log files | View test history on testgrid

Show 880 Passed Tests

Show 6515 Skipped Tests

Result	FAILURE
Tests	4 failed / 880 succeeded
Started	2023-03-11 03:55
Elapsed	55m47s
Revision	master