e2e-kops-aws-external-dns #1634681956304162816 Results

Test Failures

Kubernetes e2e suite [It] [sig-cli] Kubectl client Kubectl describe should check if kubectl describe prints relevant information for rc and pods [Conformance] 10s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-cli\]\sKubectl\sclient\sKubectl\sdescribe\sshould\scheck\sif\skubectl\sdescribe\sprints\srelevant\sinformation\sfor\src\sand\spods\s\s\[Conformance\]$'

[FAILED] error running /home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-4831 create -f -:
Command stdout:

stderr:
Unable to connect to the server: stream error: stream ID 1; INTERNAL_ERROR; received from peer

error:
exit status 1
In [It] at: test/e2e/framework/kubectl/builder.go:87 @ 03/11/23 22:40:09.09

				> Enter [BeforeEach] [sig-cli] Kubectl client - set up framework | framework.go:191 @ 03/11/23 22:40:06.79
STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/11/23 22:40:06.791
Mar 11 22:40:06.791: INFO: >>> kubeConfig: /root/.kube/config
STEP: Building a namespace api object, basename kubectl - test/e2e/framework/framework.go:250 @ 03/11/23 22:40:06.792
STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/11/23 22:40:07.442
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/11/23 22:40:07.868
< Exit [BeforeEach] [sig-cli] Kubectl client - set up framework | framework.go:191 @ 03/11/23 22:40:08.297 (1.506s)
> Enter [BeforeEach] [sig-cli] Kubectl client - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 22:40:08.297
< Exit [BeforeEach] [sig-cli] Kubectl client - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 22:40:08.297 (0s)
> Enter [BeforeEach] [sig-cli] Kubectl client - test/e2e/kubectl/kubectl.go:273 @ 03/11/23 22:40:08.297
< Exit [BeforeEach] [sig-cli] Kubectl client - test/e2e/kubectl/kubectl.go:273 @ 03/11/23 22:40:08.297 (0s)
> Enter [It] should check if kubectl describe prints relevant information for rc and pods  [Conformance] - test/e2e/kubectl/kubectl.go:1279 @ 03/11/23 22:40:08.297
Mar 11 22:40:08.297: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-4831 create -f -'
Mar 11 22:40:09.089: INFO: rc: 1
Mar 11 22:40:09.090: INFO: Unexpected error: 
    <exec.CodeExitError>: {
        Err: <*errors.errorString | 0xc00114c0a0>{
            s: "error running /home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-4831 create -f -:\nCommand stdout:\n\nstderr:\nUnable to connect to the server: stream error: stream ID 1; INTERNAL_ERROR; received from peer\n\nerror:\nexit status 1",
        },
        Code: 1,
    }
[FAILED] error running /home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=kubectl-4831 create -f -:
Command stdout:

stderr:
Unable to connect to the server: stream error: stream ID 1; INTERNAL_ERROR; received from peer

error:
exit status 1
In [It] at: test/e2e/framework/kubectl/builder.go:87 @ 03/11/23 22:40:09.09
< Exit [It] should check if kubectl describe prints relevant information for rc and pods  [Conformance] - test/e2e/kubectl/kubectl.go:1279 @ 03/11/23 22:40:09.09 (793ms)
> Enter [AfterEach] [sig-cli] Kubectl client - test/e2e/framework/node/init/init.go:33 @ 03/11/23 22:40:09.09
Mar 11 22:40:09.090: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
< Exit [AfterEach] [sig-cli] Kubectl client - test/e2e/framework/node/init/init.go:33 @ 03/11/23 22:40:09.541 (450ms)
> Enter [DeferCleanup (Each)] [sig-cli] Kubectl client - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 22:40:09.541
< Exit [DeferCleanup (Each)] [sig-cli] Kubectl client - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 22:40:09.541 (0s)
> Enter [DeferCleanup (Each)] [sig-cli] Kubectl client - dump namespaces | framework.go:209 @ 03/11/23 22:40:09.541
STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 22:40:09.541
STEP: Collecting events from namespace "kubectl-4831". - test/e2e/framework/debug/dump.go:42 @ 03/11/23 22:40:09.541
STEP: Found 0 events. - test/e2e/framework/debug/dump.go:46 @ 03/11/23 22:40:09.755
Mar 11 22:40:09.972: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Mar 11 22:40:09.972: INFO: 
Mar 11 22:40:10.187: INFO: 
Logging node info for node i-0018d62d281b1a944
Mar 11 22:40:10.401: INFO: Node Info: &Node{ObjectMeta:{i-0018d62d281b1a944    cfdd8574-b9b7-4893-b669-f8e992aa4a0e 1435 0 2023-03-11 22:36:49 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0018d62d281b1a944 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0018d62d281b1a944"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-03-11 22:37:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0018d62d281b1a944,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027199488 0} {<nil>} 3932812Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922341888 0} {<nil>} 3830412Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 22:37:50 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 22:37:50 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 22:37:50 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 22:37:50 +0000 UTC,LastTransitionTime:2023-03-11 22:37:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.186,},NodeAddress{Type:ExternalIP,Address:54.254.207.62,},NodeAddress{Type:InternalDNS,Address:i-0018d62d281b1a944.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0018d62d281b1a944.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-207-62.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25280cf0ec049b9b05ea74b6bddd69,SystemUUID:ec25280c-f0ec-049b-9b05-ea74b6bddd69,BootID:d11e51d8-3003-42dc-be7c-5013a4fdf87b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,KubeProxyVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:60946786,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 22:40:10.401: INFO: 
Logging kubelet events for node i-0018d62d281b1a944
Mar 11 22:40:10.626: INFO: 
Logging pods the kubelet thinks is on node i-0018d62d281b1a944
Mar 11 22:40:10.866: INFO: ebs-csi-node-7jb28 started at 2023-03-11 22:36:50 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:40:10.866: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:40:10.866: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:40:10.866: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:40:10.866: INFO: client-containers-37d7de55-32e7-43c7-9f48-6318bebc45e4 started at 2023-03-11 22:40:08 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:10.866: INFO: 	Container agnhost-container ready: false, restart count 0
Mar 11 22:40:10.866: INFO: downwardapi-volume-dd38a136-5cfd-4584-a0ef-b3a383799ea7 started at 2023-03-11 22:40:08 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:10.866: INFO: 	Container client-container ready: false, restart count 0
Mar 11 22:40:10.866: INFO: deployment-shared-map-item-removal-5c89cddd56-pkj7b started at 2023-03-11 22:40:08 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:10.866: INFO: 	Container nginx ready: false, restart count 0
Mar 11 22:40:10.866: INFO: metadata-volume-bfc5eb2d-3994-40ef-8e50-04335c06e79a started at 2023-03-11 22:40:08 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:10.866: INFO: 	Container client-container ready: false, restart count 0
Mar 11 22:40:10.866: INFO: externalname-service-rwlhd started at 2023-03-11 22:40:08 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:10.866: INFO: 	Container externalname-service ready: false, restart count 0
Mar 11 22:40:10.866: INFO: downwardapi-volume-c5df1671-a418-47b2-8c2d-1669866d4893 started at 2023-03-11 22:40:10 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:10.866: INFO: 	Container client-container ready: false, restart count 0
Mar 11 22:40:10.866: INFO: cilium-zv9z8 started at 2023-03-11 22:36:50 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:40:10.866: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 22:40:10.866: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 11 22:40:11.550: INFO: 
Latency metrics for node i-0018d62d281b1a944
Mar 11 22:40:11.550: INFO: 
Logging node info for node i-091642c82a7eddb0e
Mar 11 22:40:11.769: INFO: Node Info: &Node{ObjectMeta:{i-091642c82a7eddb0e    3c261825-5a9d-400e-bf5d-53c07d5c0bd1 1430 0 2023-03-11 22:36:49 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-091642c82a7eddb0e kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-091642c82a7eddb0e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-03-11 22:37:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-091642c82a7eddb0e,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027191296 0} {<nil>} 3932804Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922333696 0} {<nil>} 3830404Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 22:37:50 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 22:37:50 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 22:37:50 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 22:37:50 +0000 UTC,LastTransitionTime:2023-03-11 22:37:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.181,},NodeAddress{Type:ExternalIP,Address:54.179.156.160,},NodeAddress{Type:InternalDNS,Address:i-091642c82a7eddb0e.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-091642c82a7eddb0e.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-179-156-160.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27d32cf73f1b00b3ac0095c7778d9c,SystemUUID:ec27d32c-f73f-1b00-b3ac-0095c7778d9c,BootID:18c1c9d5-0ef1-4618-a437-4cf7cc76878a,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,KubeProxyVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:60946786,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 22:40:11.769: INFO: 
Logging kubelet events for node i-091642c82a7eddb0e
Mar 11 22:40:11.991: INFO: 
Logging pods the kubelet thinks is on node i-091642c82a7eddb0e
Mar 11 22:40:12.229: INFO: dns-test-53c00c57-fd04-4f54-aac3-c72785701da3 started at 2023-03-11 22:40:08 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:40:12.230: INFO: 	Container jessie-querier ready: false, restart count 0
Mar 11 22:40:12.230: INFO: 	Container querier ready: false, restart count 0
Mar 11 22:40:12.230: INFO: 	Container webserver ready: false, restart count 0
Mar 11 22:40:12.230: INFO: deployment-shared-map-item-removal-5c89cddd56-cdfrg started at 2023-03-11 22:40:08 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:12.230: INFO: 	Container nginx ready: false, restart count 0
Mar 11 22:40:12.230: INFO: ebs-csi-node-gjxkc started at 2023-03-11 22:36:49 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:40:12.230: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:40:12.230: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:40:12.230: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:40:12.230: INFO: coredns-745bddcfbb-pj9zs started at 2023-03-11 22:37:11 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:12.230: INFO: 	Container coredns ready: true, restart count 0
Mar 11 22:40:12.230: INFO: busybox-user-65534-c12396f6-1207-4e06-895d-580c04a119c9 started at 2023-03-11 22:40:07 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:12.230: INFO: 	Container busybox-user-65534-c12396f6-1207-4e06-895d-580c04a119c9 ready: false, restart count 0
Mar 11 22:40:12.230: INFO: security-context-8dd2816b-28f8-4526-a5f9-55af9850bdf3 started at 2023-03-11 22:40:08 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:12.230: INFO: 	Container test-container ready: false, restart count 0
Mar 11 22:40:12.230: INFO: cilium-fsnfv started at 2023-03-11 22:36:49 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:40:12.230: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 22:40:12.230: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 11 22:40:12.230: INFO: httpd started at 2023-03-11 22:40:11 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:12.230: INFO: 	Container httpd ready: false, restart count 0
Mar 11 22:40:12.230: INFO: update-demo-nautilus-xw6b2 started at 2023-03-11 22:40:11 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:12.230: INFO: 	Container update-demo ready: false, restart count 0
Mar 11 22:40:12.922: INFO: 
Latency metrics for node i-091642c82a7eddb0e
Mar 11 22:40:12.922: INFO: 
Logging node info for node i-0a2163d12c183f215
Mar 11 22:40:13.137: INFO: Node Info: &Node{ObjectMeta:{i-0a2163d12c183f215    7102c397-377c-4dde-9536-441cf78d33c9 1364 0 2023-03-11 22:36:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0a2163d12c183f215 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0a2163d12c183f215"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 22:36:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:36:31 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 22:36:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-11 22:36:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:36:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 22:37:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0a2163d12c183f215,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027191296 0} {<nil>} 3932804Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922333696 0} {<nil>} 3830404Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 22:37:32 +0000 UTC,LastTransitionTime:2023-03-11 22:36:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 22:37:32 +0000 UTC,LastTransitionTime:2023-03-11 22:36:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 22:37:32 +0000 UTC,LastTransitionTime:2023-03-11 22:36:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 22:37:32 +0000 UTC,LastTransitionTime:2023-03-11 22:36:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.63.42,},NodeAddress{Type:ExternalIP,Address:46.137.231.215,},NodeAddress{Type:InternalDNS,Address:i-0a2163d12c183f215.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0a2163d12c183f215.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-46-137-231-215.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec22c6e1f6d754b17f85187d7d62103d,SystemUUID:ec22c6e1-f6d7-54b1-7f85-187d7d62103d,BootID:856c0a60-d0bf-4c9a-b89e-ad079db9a26a,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,KubeProxyVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:60946786,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 22:40:13.137: INFO: 
Logging kubelet events for node i-0a2163d12c183f215
Mar 11 22:40:13.356: INFO: 
Logging pods the kubelet thinks is on node i-0a2163d12c183f215
Mar 11 22:40:13.594: INFO: cilium-tsxzj started at 2023-03-11 22:36:32 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:40:13.594: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 22:40:13.594: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 11 22:40:13.594: INFO: ebs-csi-node-g7wxp started at 2023-03-11 22:36:32 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:40:13.594: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:40:13.594: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:40:13.594: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:40:13.594: INFO: coredns-745bddcfbb-ldcr7 started at 2023-03-11 22:36:43 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:13.594: INFO: 	Container coredns ready: true, restart count 0
Mar 11 22:40:13.594: INFO: coredns-autoscaler-8495f5575-bj25x started at 2023-03-11 22:36:43 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:13.594: INFO: 	Container autoscaler ready: true, restart count 0
Mar 11 22:40:13.594: INFO: pod-subpath-test-inlinevolume-5j8w started at 2023-03-11 22:40:09 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:40:13.594: INFO: 	Init container init-volume-inlinevolume-5j8w ready: false, restart count 0
Mar 11 22:40:13.594: INFO: 	Container test-container-subpath-inlinevolume-5j8w ready: false, restart count 0
Mar 11 22:40:13.594: INFO: hostexec-i-0a2163d12c183f215-wjk74 started at 2023-03-11 22:40:12 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:13.594: INFO: 	Container agnhost-container ready: false, restart count 0
Mar 11 22:40:13.594: INFO: ebs-csi-controller-56487f47bb-hm785 started at 2023-03-11 22:36:43 +0000 UTC (0+5 container statuses recorded)
Mar 11 22:40:13.594: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:40:13.594: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:40:13.594: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 22:40:13.594: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:40:13.594: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:40:13.594: INFO: externalname-service-6xq7f started at 2023-03-11 22:40:08 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:13.594: INFO: 	Container externalname-service ready: true, restart count 0
Mar 11 22:40:13.594: INFO: deployment-shared-map-item-removal-5c89cddd56-f7j8x started at 2023-03-11 22:40:08 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:13.594: INFO: 	Container nginx ready: false, restart count 0
Mar 11 22:40:14.298: INFO: 
Latency metrics for node i-0a2163d12c183f215
Mar 11 22:40:14.298: INFO: 
Logging node info for node i-0ceedc6162dac7c42
Mar 11 22:40:14.512: INFO: Node Info: &Node{ObjectMeta:{i-0ceedc6162dac7c42    adea5b88-823a-4c79-bfd5-3e46a8aaf8f4 1420 0 2023-03-11 22:36:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0ceedc6162dac7c42 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0ceedc6162dac7c42"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 22:36:46 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:36:46 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 22:36:46 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:36:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 22:36:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-03-11 22:37:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0ceedc6162dac7c42,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027187200 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922329600 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 22:37:47 +0000 UTC,LastTransitionTime:2023-03-11 22:36:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 22:37:47 +0000 UTC,LastTransitionTime:2023-03-11 22:36:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 22:37:47 +0000 UTC,LastTransitionTime:2023-03-11 22:36:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 22:37:47 +0000 UTC,LastTransitionTime:2023-03-11 22:36:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.91,},NodeAddress{Type:ExternalIP,Address:52.77.247.227,},NodeAddress{Type:InternalDNS,Address:i-0ceedc6162dac7c42.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ceedc6162dac7c42.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-77-247-227.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2956a0e70ff2db5f15382b6e61f0c7,SystemUUID:ec2956a0-e70f-f2db-5f15-382b6e61f0c7,BootID:5df332c2-10b8-4fd4-a70f-4b257ccfb865,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,KubeProxyVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:60946786,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 22:40:14.513: INFO: 
Logging kubelet events for node i-0ceedc6162dac7c42
Mar 11 22:40:14.731: INFO: 
Logging pods the kubelet thinks is on node i-0ceedc6162dac7c42
Mar 11 22:40:14.970: INFO: cilium-bhp52 started at 2023-03-11 22:36:47 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:40:14.970: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 22:40:14.970: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 11 22:40:14.970: INFO: ebs-csi-node-tk8q8 started at 2023-03-11 22:36:47 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:40:14.970: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:40:14.970: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:40:14.970: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:40:14.970: INFO: ebs-csi-controller-56487f47bb-qbsdk started at 2023-03-11 22:36:57 +0000 UTC (0+5 container statuses recorded)
Mar 11 22:40:14.970: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:40:14.970: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:40:14.970: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 22:40:14.970: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:40:14.970: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:40:14.970: INFO: pod-projected-secrets-02c1ce3c-2fc6-41ad-8f5b-277dc231d2bc started at 2023-03-11 22:40:08 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:14.970: INFO: 	Container projected-secret-volume-test ready: false, restart count 0
Mar 11 22:40:14.970: INFO: sample-webhook-deployment-7497495989-q6nfh started at 2023-03-11 22:40:10 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:14.970: INFO: 	Container sample-webhook ready: true, restart count 0
Mar 11 22:40:14.970: INFO: agnhost started at 2023-03-11 22:40:10 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:14.970: INFO: 	Container agnhost ready: false, restart count 0
Mar 11 22:40:15.680: INFO: 
Latency metrics for node i-0ceedc6162dac7c42
Mar 11 22:40:15.680: INFO: 
Logging node info for node i-0ef539cea8af5adb9
Mar 11 22:40:15.899: INFO: Node Info: &Node{ObjectMeta:{i-0ef539cea8af5adb9    b247edc6-7a24-457b-85a5-35f8a510c12c 898 0 2023-03-11 22:32:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0ef539cea8af5adb9 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0ef539cea8af5adb9"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 22:32:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 22:32:57 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:33:39 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:33:39 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 22:33:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 22:36:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0ef539cea8af5adb9,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4002074624 0} {<nil>} 3908276Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3897217024 0} {<nil>} 3805876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 22:36:03 +0000 UTC,LastTransitionTime:2023-03-11 22:32:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 22:36:03 +0000 UTC,LastTransitionTime:2023-03-11 22:32:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 22:36:03 +0000 UTC,LastTransitionTime:2023-03-11 22:32:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 22:36:03 +0000 UTC,LastTransitionTime:2023-03-11 22:33:39 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.218,},NodeAddress{Type:ExternalIP,Address:54.169.229.126,},NodeAddress{Type:InternalDNS,Address:i-0ef539cea8af5adb9.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ef539cea8af5adb9.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-169-229-126.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2b39ada76eefe97ffcfe201a7e5f77,SystemUUID:ec2b39ad-a76e-efe9-7ffc-fe201a7e5f77,BootID:e1af46ce-9267-4c6e-9738-fdc23cbf769f,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,KubeProxyVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:115123555,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:107914749,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:60946786,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:56731120,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41885191,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40376038,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/external-dns/external-dns@sha256:1529982e95d0c9a1f0543360939108cf8276507fb6639e367a0198d3b25570b0 registry.k8s.io/external-dns/external-dns:v0.13.1],SizeBytes:22414358,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740762,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 22:40:15.900: INFO: 
Logging kubelet events for node i-0ef539cea8af5adb9
Mar 11 22:40:16.129: INFO: 
Logging pods the kubelet thinks is on node i-0ef539cea8af5adb9
Mar 11 22:40:16.384: INFO: etcd-manager-main-i-0ef539cea8af5adb9 started at 2023-03-11 22:30:24 +0000 UTC (11+1 container statuses recorded)
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 11 22:40:16.384: INFO: kube-apiserver-i-0ef539cea8af5adb9 started at 2023-03-11 22:30:24 +0000 UTC (0+2 container statuses recorded)
Mar 11 22:40:16.384: INFO: 	Container healthcheck ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Container kube-apiserver ready: true, restart count 3
Mar 11 22:40:16.384: INFO: kube-scheduler-i-0ef539cea8af5adb9 started at 2023-03-11 22:30:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:16.384: INFO: 	Container kube-scheduler ready: true, restart count 0
Mar 11 22:40:16.384: INFO: aws-cloud-controller-manager-jn2dh started at 2023-03-11 22:33:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:16.384: INFO: 	Container aws-cloud-controller-manager ready: true, restart count 0
Mar 11 22:40:16.384: INFO: cilium-operator-66756dbf95-f9895 started at 2023-03-11 22:33:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:16.384: INFO: 	Container cilium-operator ready: true, restart count 1
Mar 11 22:40:16.384: INFO: cilium-rsmft started at 2023-03-11 22:33:24 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:40:16.384: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Container cilium-agent ready: true, restart count 1
Mar 11 22:40:16.384: INFO: external-dns-5d96c7b864-f8tgk started at 2023-03-11 22:33:39 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:16.384: INFO: 	Container external-dns ready: true, restart count 0
Mar 11 22:40:16.384: INFO: etcd-manager-events-i-0ef539cea8af5adb9 started at 2023-03-11 22:30:24 +0000 UTC (11+1 container statuses recorded)
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 11 22:40:16.384: INFO: ebs-csi-node-8ztn8 started at 2023-03-11 22:33:24 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:40:16.384: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:40:16.384: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:40:16.384: INFO: kops-controller-4b6hj started at 2023-03-11 22:33:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:16.384: INFO: 	Container kops-controller ready: true, restart count 0
Mar 11 22:40:16.384: INFO: kube-controller-manager-i-0ef539cea8af5adb9 started at 2023-03-11 22:30:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:40:16.384: INFO: 	Container kube-controller-manager ready: true, restart count 4
Mar 11 22:40:17.112: INFO: 
Latency metrics for node i-0ef539cea8af5adb9
END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 22:40:17.112 (7.571s)
< Exit [DeferCleanup (Each)] [sig-cli] Kubectl client - dump namespaces | framework.go:209 @ 03/11/23 22:40:17.112 (7.571s)
> Enter [DeferCleanup (Each)] [sig-cli] Kubectl client - tear down framework | framework.go:206 @ 03/11/23 22:40:17.112
STEP: Destroying namespace "kubectl-4831" for this suite. - test/e2e/framework/framework.go:351 @ 03/11/23 22:40:17.112
< Exit [DeferCleanup (Each)] [sig-cli] Kubectl client - tear down framework | framework.go:206 @ 03/11/23 22:40:17.33 (218ms)
> Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 22:40:17.33
< Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 22:40:17.33 (0s)

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

Kubernetes e2e suite [It] [sig-network] EndpointSlice should support a Service with multiple ports specified in multiple EndpointSlices 2m34s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-network\]\sEndpointSlice\sshould\ssupport\sa\sService\swith\smultiple\sports\sspecified\sin\smultiple\sEndpointSlices$'

[FAILED] error running /home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-8754 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.68.96:80/hostname:
Command stdout:

stderr:
+ curl -q -s --connect-timeout 30 100.65.68.96:80/hostname
command terminated with exit code 28

error:
exit status 28
In [It] at: test/e2e/network/util.go:180 @ 03/11/23 22:51:00.516

				> Enter [BeforeEach] [sig-network] EndpointSlice - set up framework | framework.go:191 @ 03/11/23 22:48:36.068
STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/11/23 22:48:36.068
Mar 11 22:48:36.068: INFO: >>> kubeConfig: /root/.kube/config
STEP: Building a namespace api object, basename endpointslice - test/e2e/framework/framework.go:250 @ 03/11/23 22:48:36.07
STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/11/23 22:48:36.715
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/11/23 22:48:37.142
< Exit [BeforeEach] [sig-network] EndpointSlice - set up framework | framework.go:191 @ 03/11/23 22:48:37.569 (1.501s)
> Enter [BeforeEach] [sig-network] EndpointSlice - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 22:48:37.569
< Exit [BeforeEach] [sig-network] EndpointSlice - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 22:48:37.569 (0s)
> Enter [BeforeEach] [sig-network] EndpointSlice - test/e2e/network/endpointslice.go:54 @ 03/11/23 22:48:37.569
< Exit [BeforeEach] [sig-network] EndpointSlice - test/e2e/network/endpointslice.go:54 @ 03/11/23 22:48:37.569 (0s)
> Enter [It] should support a Service with multiple ports specified in multiple EndpointSlices - test/e2e/network/endpointslice.go:530 @ 03/11/23 22:48:37.569
STEP: creating - test/e2e/network/endpointslice.go:597 @ 03/11/23 22:48:40.672
STEP: Creating a pause pods that will try to connect to the webserver - test/e2e/network/endpointslice.go:618 @ 03/11/23 22:48:41.128
Mar 11 22:48:43.992: INFO: Waiting up to 2m0s to get response from 100.65.68.96:80
Mar 11 22:48:43.992: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-8754 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.68.96:80/hostname'
Mar 11 22:49:16.123: INFO: rc: 28
Mar 11 22:49:16.123: INFO: got err: error running /home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-8754 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.68.96:80/hostname:
Command stdout:

stderr:
+ curl -q -s --connect-timeout 30 100.65.68.96:80/hostname
command terminated with exit code 28

error:
exit status 28, retry until timeout
Mar 11 22:49:18.124: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-8754 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.68.96:80/hostname'
Mar 11 22:49:50.259: INFO: rc: 28
Mar 11 22:49:50.259: INFO: got err: error running /home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-8754 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.68.96:80/hostname:
Command stdout:

stderr:
+ curl -q -s --connect-timeout 30 100.65.68.96:80/hostname
command terminated with exit code 28

error:
exit status 28, retry until timeout
Mar 11 22:49:52.259: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-8754 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.68.96:80/hostname'
Mar 11 22:50:24.411: INFO: rc: 28
Mar 11 22:50:24.411: INFO: got err: error running /home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-8754 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.68.96:80/hostname:
Command stdout:

stderr:
+ curl -q -s --connect-timeout 30 100.65.68.96:80/hostname
command terminated with exit code 28

error:
exit status 28, retry until timeout
Mar 11 22:50:26.411: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-8754 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.68.96:80/hostname'
Mar 11 22:50:58.515: INFO: rc: 28
Mar 11 22:50:58.515: INFO: got err: error running /home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-8754 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.68.96:80/hostname:
Command stdout:

stderr:
+ curl -q -s --connect-timeout 30 100.65.68.96:80/hostname
command terminated with exit code 28

error:
exit status 28, retry until timeout
Mar 11 22:51:00.516: INFO: Unexpected error: 
    <exec.CodeExitError>: {
        Err: <*errors.errorString | 0xc001192170>{
            s: "error running /home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-8754 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.68.96:80/hostname:\nCommand stdout:\n\nstderr:\n+ curl -q -s --connect-timeout 30 100.65.68.96:80/hostname\ncommand terminated with exit code 28\n\nerror:\nexit status 28",
        },
        Code: 28,
    }
[FAILED] error running /home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=endpointslice-8754 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 100.65.68.96:80/hostname:
Command stdout:

stderr:
+ curl -q -s --connect-timeout 30 100.65.68.96:80/hostname
command terminated with exit code 28

error:
exit status 28
In [It] at: test/e2e/network/util.go:180 @ 03/11/23 22:51:00.516
< Exit [It] should support a Service with multiple ports specified in multiple EndpointSlices - test/e2e/network/endpointslice.go:530 @ 03/11/23 22:51:00.516 (2m22.947s)
> Enter [AfterEach] [sig-network] EndpointSlice - test/e2e/framework/node/init/init.go:33 @ 03/11/23 22:51:00.516
Mar 11 22:51:00.516: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
< Exit [AfterEach] [sig-network] EndpointSlice - test/e2e/framework/node/init/init.go:33 @ 03/11/23 22:51:01.65 (1.134s)
> Enter [DeferCleanup (Each)] [sig-network] EndpointSlice - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 22:51:01.65
< Exit [DeferCleanup (Each)] [sig-network] EndpointSlice - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 22:51:01.65 (0s)
> Enter [DeferCleanup (Each)] [sig-network] EndpointSlice - dump namespaces | framework.go:209 @ 03/11/23 22:51:01.65
STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 22:51:01.65
STEP: Collecting events from namespace "endpointslice-8754". - test/e2e/framework/debug/dump.go:42 @ 03/11/23 22:51:01.65
STEP: Found 11 events. - test/e2e/framework/debug/dump.go:46 @ 03/11/23 22:51:01.867
Mar 11 22:51:01.867: INFO: At 2023-03-11 22:48:37 +0000 UTC - event for pod-handle-http-request: {default-scheduler } Scheduled: Successfully assigned endpointslice-8754/pod-handle-http-request to i-0018d62d281b1a944
Mar 11 22:51:01.867: INFO: At 2023-03-11 22:48:38 +0000 UTC - event for pod-handle-http-request: {kubelet i-0018d62d281b1a944} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Mar 11 22:51:01.867: INFO: At 2023-03-11 22:48:38 +0000 UTC - event for pod-handle-http-request: {kubelet i-0018d62d281b1a944} Created: Created container container-handle-8090-request
Mar 11 22:51:01.867: INFO: At 2023-03-11 22:48:38 +0000 UTC - event for pod-handle-http-request: {kubelet i-0018d62d281b1a944} Started: Started container container-handle-8090-request
Mar 11 22:51:01.867: INFO: At 2023-03-11 22:48:38 +0000 UTC - event for pod-handle-http-request: {kubelet i-0018d62d281b1a944} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Mar 11 22:51:01.867: INFO: At 2023-03-11 22:48:38 +0000 UTC - event for pod-handle-http-request: {kubelet i-0018d62d281b1a944} Created: Created container container-handle-9090-request
Mar 11 22:51:01.867: INFO: At 2023-03-11 22:48:38 +0000 UTC - event for pod-handle-http-request: {kubelet i-0018d62d281b1a944} Started: Started container container-handle-9090-request
Mar 11 22:51:01.867: INFO: At 2023-03-11 22:48:41 +0000 UTC - event for pause-pod-0: {default-scheduler } Scheduled: Successfully assigned endpointslice-8754/pause-pod-0 to i-091642c82a7eddb0e
Mar 11 22:51:01.867: INFO: At 2023-03-11 22:48:41 +0000 UTC - event for pause-pod-0: {kubelet i-091642c82a7eddb0e} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Mar 11 22:51:01.867: INFO: At 2023-03-11 22:48:42 +0000 UTC - event for pause-pod-0: {kubelet i-091642c82a7eddb0e} Created: Created container agnhost-container
Mar 11 22:51:01.867: INFO: At 2023-03-11 22:48:42 +0000 UTC - event for pause-pod-0: {kubelet i-091642c82a7eddb0e} Started: Started container agnhost-container
Mar 11 22:51:02.083: INFO: POD                      NODE                 PHASE    GRACE  CONDITIONS
Mar 11 22:51:02.083: INFO: pause-pod-0              i-091642c82a7eddb0e  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:48:41 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:48:42 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:48:42 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:48:41 +0000 UTC  }]
Mar 11 22:51:02.083: INFO: pod-handle-http-request  i-0018d62d281b1a944  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:48:37 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:48:39 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:48:39 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:48:37 +0000 UTC  }]
Mar 11 22:51:02.083: INFO: 
Mar 11 22:51:02.985: INFO: 
Logging node info for node i-0018d62d281b1a944
Mar 11 22:51:03.205: INFO: Node Info: &Node{ObjectMeta:{i-0018d62d281b1a944    cfdd8574-b9b7-4893-b669-f8e992aa4a0e 32513 0 2023-03-11 22:36:49 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0018d62d281b1a944 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0018d62d281b1a944 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1356":"i-0018d62d281b1a944","csi-hostpath-provisioning-4570":"i-0018d62d281b1a944","ebs.csi.aws.com":"i-0018d62d281b1a944"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:50:58 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-11 22:50:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0018d62d281b1a944,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027199488 0} {<nil>} 3932812Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922341888 0} {<nil>} 3830412Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 22:50:58 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 22:50:58 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 22:50:58 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 22:50:58 +0000 UTC,LastTransitionTime:2023-03-11 22:37:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.186,},NodeAddress{Type:ExternalIP,Address:54.254.207.62,},NodeAddress{Type:InternalDNS,Address:i-0018d62d281b1a944.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0018d62d281b1a944.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-207-62.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25280cf0ec049b9b05ea74b6bddd69,SystemUUID:ec25280c-f0ec-049b-9b05-ea74b6bddd69,BootID:d11e51d8-3003-42dc-be7c-5013a4fdf87b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,KubeProxyVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:60946786,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-1356^1cbce745-c05f-11ed-8154-a6c41588a4f7 kubernetes.io/csi/csi-hostpath-ephemeral-1356^1f341ae9-c05f-11ed-8154-a6c41588a4f7],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-1356^1cbce745-c05f-11ed-8154-a6c41588a4f7,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-1356^1f341ae9-c05f-11ed-8154-a6c41588a4f7,DevicePath:,},},Config:nil,},}
Mar 11 22:51:03.205: INFO: 
Logging kubelet events for node i-0018d62d281b1a944
Mar 11 22:51:03.492: INFO: 
Logging pods the kubelet thinks is on node i-0018d62d281b1a944
Mar 11 22:51:03.931: INFO: webserver-67bd4bf6dc-rhjvg started at 2023-03-11 22:51:02 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:03.931: INFO: 	Container httpd ready: false, restart count 0
Mar 11 22:51:03.931: INFO: pod-handle-http-request started at 2023-03-11 22:48:37 +0000 UTC (0+2 container statuses recorded)
Mar 11 22:51:03.931: INFO: 	Container container-handle-8090-request ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container container-handle-9090-request ready: true, restart count 0
Mar 11 22:51:03.931: INFO: csi-hostpathplugin-0 started at 2023-03-11 22:50:23 +0000 UTC (0+7 container statuses recorded)
Mar 11 22:51:03.931: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container csi-snapshotter ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container hostpath ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:51:03.931: INFO: liveness-8c109247-cf83-44df-8e22-5e74faeaf481 started at 2023-03-11 22:48:01 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:03.931: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 22:51:03.931: INFO: webserver-pod started at 2023-03-11 22:46:49 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:03.931: INFO: 	Container agnhost ready: false, restart count 0
Mar 11 22:51:03.931: INFO: ebs-csi-node-7jb28 started at 2023-03-11 22:36:50 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:51:03.931: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:51:03.931: INFO: test-rs-hvgqv started at 2023-03-11 22:50:59 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:03.931: INFO: 	Container httpd ready: true, restart count 0
Mar 11 22:51:03.931: INFO: inline-volume-tester-jdwgb started at 2023-03-11 22:50:26 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:03.931: INFO: 	Container csi-volume-tester ready: true, restart count 0
Mar 11 22:51:03.931: INFO: cilium-zv9z8 started at 2023-03-11 22:36:50 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:51:03.931: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 11 22:51:03.931: INFO: csi-hostpathplugin-0 started at 2023-03-11 22:49:58 +0000 UTC (0+7 container statuses recorded)
Mar 11 22:51:03.931: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container csi-snapshotter ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container hostpath ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:51:03.931: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:51:03.931: INFO: netserver-0 started at 2023-03-11 22:50:21 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:03.931: INFO: 	Container webserver ready: true, restart count 0
Mar 11 22:51:03.931: INFO: affinity-nodeport-timeout-whsd2 started at 2023-03-11 22:50:26 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:03.931: INFO: 	Container affinity-nodeport-timeout ready: true, restart count 0
Mar 11 22:51:03.931: INFO: pod-qos-class-c0781a78-a18c-4e5b-bffc-e7624a94fad1 started at 2023-03-11 22:50:59 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:03.931: INFO: 	Container agnhost ready: true, restart count 0
Mar 11 22:51:03.931: INFO: inline-volume-tester2-v7d9x started at 2023-03-11 22:50:31 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:03.931: INFO: 	Container csi-volume-tester ready: true, restart count 0
Mar 11 22:51:04.727: INFO: 
Latency metrics for node i-0018d62d281b1a944
Mar 11 22:51:04.727: INFO: 
Logging node info for node i-091642c82a7eddb0e
Mar 11 22:51:04.944: INFO: Node Info: &Node{ObjectMeta:{i-091642c82a7eddb0e    3c261825-5a9d-400e-bf5d-53c07d5c0bd1 31814 0 2023-03-11 22:36:49 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-091642c82a7eddb0e kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-091642c82a7eddb0e topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-expansion-588":"i-091642c82a7eddb0e","ebs.csi.aws.com":"i-091642c82a7eddb0e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-03-11 22:50:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-091642c82a7eddb0e,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027191296 0} {<nil>} 3932804Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922333696 0} {<nil>} 3830404Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 22:50:16 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 22:50:16 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 22:50:16 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 22:50:16 +0000 UTC,LastTransitionTime:2023-03-11 22:37:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.181,},NodeAddress{Type:ExternalIP,Address:54.179.156.160,},NodeAddress{Type:InternalDNS,Address:i-091642c82a7eddb0e.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-091642c82a7eddb0e.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-179-156-160.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27d32cf73f1b00b3ac0095c7778d9c,SystemUUID:ec27d32c-f73f-1b00-b3ac-0095c7778d9c,BootID:18c1c9d5-0ef1-4618-a437-4cf7cc76878a,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,KubeProxyVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:60946786,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:21331383,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 22:51:04.944: INFO: 
Logging kubelet events for node i-091642c82a7eddb0e
Mar 11 22:51:05.172: INFO: 
Logging pods the kubelet thinks is on node i-091642c82a7eddb0e
Mar 11 22:51:05.395: INFO: pvc-volume-tester-mf42g started at 2023-03-11 22:50:46 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:05.395: INFO: 	Container volume-tester ready: true, restart count 0
Mar 11 22:51:05.395: INFO: coredns-745bddcfbb-pj9zs started at 2023-03-11 22:37:11 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:05.395: INFO: 	Container coredns ready: true, restart count 0
Mar 11 22:51:05.395: INFO: execpod-affinity48vzv started at 2023-03-11 22:50:30 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:05.395: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 22:51:05.395: INFO: affinity-nodeport-timeout-5wsbw started at 2023-03-11 22:50:26 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:05.395: INFO: 	Container affinity-nodeport-timeout ready: true, restart count 0
Mar 11 22:51:05.395: INFO: pause-pod-0 started at 2023-03-11 22:48:41 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:05.395: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 22:51:05.395: INFO: csi-mockplugin-resizer-0 started at 2023-03-11 22:50:54 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:05.395: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 22:51:05.395: INFO: ebs-csi-node-gjxkc started at 2023-03-11 22:36:49 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:51:05.395: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:51:05.395: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:51:05.395: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:51:05.395: INFO: csi-mockplugin-0 started at 2023-03-11 22:50:53 +0000 UTC (0+4 container statuses recorded)
Mar 11 22:51:05.395: INFO: 	Container busybox ready: true, restart count 0
Mar 11 22:51:05.395: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:51:05.395: INFO: 	Container driver-registrar ready: true, restart count 0
Mar 11 22:51:05.395: INFO: 	Container mock ready: true, restart count 0
Mar 11 22:51:05.395: INFO: cilium-fsnfv started at 2023-03-11 22:36:49 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:51:05.395: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 22:51:05.395: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 11 22:51:05.395: INFO: csi-mockplugin-attacher-0 started at 2023-03-11 22:50:53 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:05.395: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:51:05.395: INFO: csi-mockplugin-0 started at 2023-03-11 22:50:41 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:51:05.395: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:51:05.395: INFO: 	Container driver-registrar ready: true, restart count 0
Mar 11 22:51:05.395: INFO: 	Container mock ready: true, restart count 0
Mar 11 22:51:05.395: INFO: csi-mockplugin-resizer-0 started at 2023-03-11 22:50:42 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:05.395: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 22:51:06.191: INFO: 
Latency metrics for node i-091642c82a7eddb0e
Mar 11 22:51:06.191: INFO: 
Logging node info for node i-0a2163d12c183f215
Mar 11 22:51:06.414: INFO: Node Info: &Node{ObjectMeta:{i-0a2163d12c183f215    7102c397-377c-4dde-9536-441cf78d33c9 33016 0 2023-03-11 22:36:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0a2163d12c183f215 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0a2163d12c183f215 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-5465":"i-0a2163d12c183f215","csi-hostpath-volume-expand-9913":"i-0a2163d12c183f215","ebs.csi.aws.com":"i-0a2163d12c183f215"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 22:36:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:36:31 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 22:36:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-11 22:36:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:36:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:51:01 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-11 22:51:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0a2163d12c183f215,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027191296 0} {<nil>} 3932804Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922333696 0} {<nil>} 3830404Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 22:50:37 +0000 UTC,LastTransitionTime:2023-03-11 22:36:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 22:50:37 +0000 UTC,LastTransitionTime:2023-03-11 22:36:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 22:50:37 +0000 UTC,LastTransitionTime:2023-03-11 22:36:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 22:50:37 +0000 UTC,LastTransitionTime:2023-03-11 22:36:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.63.42,},NodeAddress{Type:ExternalIP,Address:46.137.231.215,},NodeAddress{Type:InternalDNS,Address:i-0a2163d12c183f215.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0a2163d12c183f215.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-46-137-231-215.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec22c6e1f6d754b17f85187d7d62103d,SystemUUID:ec22c6e1-f6d7-54b1-7f85-187d7d62103d,BootID:856c0a60-d0bf-4c9a-b89e-ad079db9a26a,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,KubeProxyVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:60946786,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0251f18a3835172a6],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-expand-9913^3058d2cd-c05f-11ed-9d3d-7ee7da543a77,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0251f18a3835172a6,DevicePath:,},},Config:nil,},}
Mar 11 22:51:06.414: INFO: 
Logging kubelet events for node i-0a2163d12c183f215
Mar 11 22:51:06.637: INFO: 
Logging pods the kubelet thinks is on node i-0a2163d12c183f215
Mar 11 22:51:07.085: INFO: pod-8d0c1339-feb0-471f-a780-d88b58e3fa18 started at 2023-03-11 22:51:00 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:07.085: INFO: 	Container write-pod ready: false, restart count 0
Mar 11 22:51:07.085: INFO: ebs-csi-node-g7wxp started at 2023-03-11 22:36:32 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:51:07.085: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:51:07.085: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:51:07.085: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:51:07.085: INFO: coredns-745bddcfbb-ldcr7 started at 2023-03-11 22:36:43 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:07.085: INFO: 	Container coredns ready: true, restart count 0
Mar 11 22:51:07.085: INFO: coredns-autoscaler-8495f5575-bj25x started at 2023-03-11 22:36:43 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:07.085: INFO: 	Container autoscaler ready: true, restart count 0
Mar 11 22:51:07.085: INFO: hostexec-i-0a2163d12c183f215-9plcd started at 2023-03-11 22:50:48 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:07.085: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 22:51:07.085: INFO: webserver-67bd4bf6dc-wq42x started at 2023-03-11 22:51:02 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:07.085: INFO: 	Container httpd ready: true, restart count 0
Mar 11 22:51:07.085: INFO: csi-hostpathplugin-0 started at 2023-03-11 22:51:04 +0000 UTC (0+7 container statuses recorded)
Mar 11 22:51:07.085: INFO: 	Container csi-attacher ready: false, restart count 0
Mar 11 22:51:07.085: INFO: 	Container csi-provisioner ready: false, restart count 0
Mar 11 22:51:07.085: INFO: 	Container csi-resizer ready: false, restart count 0
Mar 11 22:51:07.085: INFO: 	Container csi-snapshotter ready: false, restart count 0
Mar 11 22:51:07.085: INFO: 	Container hostpath ready: false, restart count 0
Mar 11 22:51:07.085: INFO: 	Container liveness-probe ready: false, restart count 0
Mar 11 22:51:07.085: INFO: 	Container node-driver-registrar ready: false, restart count 0
Mar 11 22:51:07.085: INFO: cilium-tsxzj started at 2023-03-11 22:36:32 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:51:07.085: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 22:51:07.085: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 11 22:51:07.085: INFO: inline-volume-tester-hcpdc started at 2023-03-11 22:49:42 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:07.085: INFO: 	Container csi-volume-tester ready: true, restart count 0
Mar 11 22:51:07.085: INFO: csi-hostpathplugin-0 started at 2023-03-11 22:50:57 +0000 UTC (0+7 container statuses recorded)
Mar 11 22:51:07.085: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:51:07.085: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:51:07.085: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 22:51:07.085: INFO: 	Container csi-snapshotter ready: true, restart count 0
Mar 11 22:51:07.085: INFO: 	Container hostpath ready: true, restart count 0
Mar 11 22:51:07.085: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:51:07.085: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:51:07.085: INFO: pod-subpath-test-preprovisionedpv-vnnd started at 2023-03-11 22:51:02 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:51:07.085: INFO: 	Init container init-volume-preprovisionedpv-vnnd ready: true, restart count 0
Mar 11 22:51:07.085: INFO: 	Container test-container-subpath-preprovisionedpv-vnnd ready: true, restart count 0
Mar 11 22:51:07.085: INFO: ebs-csi-controller-56487f47bb-hm785 started at 2023-03-11 22:36:43 +0000 UTC (0+5 container statuses recorded)
Mar 11 22:51:07.085: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:51:07.085: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:51:07.085: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 22:51:07.085: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:51:07.085: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:51:07.858: INFO: 
Latency metrics for node i-0a2163d12c183f215
Mar 11 22:51:07.858: INFO: 
Logging node info for node i-0ceedc6162dac7c42
Mar 11 22:51:08.074: INFO: Node Info: &Node{ObjectMeta:{i-0ceedc6162dac7c42    adea5b88-823a-4c79-bfd5-3e46a8aaf8f4 32867 0 2023-03-11 22:36:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0ceedc6162dac7c42 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0ceedc6162dac7c42 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-3889":"i-0ceedc6162dac7c42","csi-mock-csi-mock-volumes-attach-8758":"i-0ceedc6162dac7c42","ebs.csi.aws.com":"i-0ceedc6162dac7c42"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 22:36:46 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:36:46 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 22:36:46 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:36:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 22:36:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:51:04 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-11 22:51:04 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0ceedc6162dac7c42,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027187200 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922329600 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 22:51:04 +0000 UTC,LastTransitionTime:2023-03-11 22:36:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 22:51:04 +0000 UTC,LastTransitionTime:2023-03-11 22:36:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 22:51:04 +0000 UTC,LastTransitionTime:2023-03-11 22:36:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 22:51:04 +0000 UTC,LastTransitionTime:2023-03-11 22:36:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.91,},NodeAddress{Type:ExternalIP,Address:52.77.247.227,},NodeAddress{Type:InternalDNS,Address:i-0ceedc6162dac7c42.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ceedc6162dac7c42.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-77-247-227.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2956a0e70ff2db5f15382b6e61f0c7,SystemUUID:ec2956a0-e70f-f2db-5f15-382b6e61f0c7,BootID:5df332c2-10b8-4fd4-a70f-4b257ccfb865,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,KubeProxyVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:60946786,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-attach-8758^2f3ed6c9-c05f-11ed-a3de-aa60041c5307],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-attach-8758^2f3ed6c9-c05f-11ed-a3de-aa60041c5307,DevicePath:,},},Config:nil,},}
Mar 11 22:51:08.074: INFO: 
Logging kubelet events for node i-0ceedc6162dac7c42
Mar 11 22:51:08.295: INFO: 
Logging pods the kubelet thinks is on node i-0ceedc6162dac7c42
Mar 11 22:51:08.519: INFO: affinity-nodeport-timeout-qqbch started at 2023-03-11 22:50:26 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:08.519: INFO: 	Container affinity-nodeport-timeout ready: true, restart count 0
Mar 11 22:51:08.519: INFO: cilium-bhp52 started at 2023-03-11 22:36:47 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:51:08.519: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 11 22:51:08.519: INFO: csi-mockplugin-0 started at 2023-03-11 22:50:53 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:51:08.519: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container driver-registrar ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container mock ready: true, restart count 0
Mar 11 22:51:08.519: INFO: csi-hostpathplugin-0 started at 2023-03-11 22:50:46 +0000 UTC (0+7 container statuses recorded)
Mar 11 22:51:08.519: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container csi-snapshotter ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container hostpath ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:51:08.519: INFO: kube-proxy-mode-detector started at 2023-03-11 22:50:21 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:08.519: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 22:51:08.519: INFO: ebs-csi-node-tk8q8 started at 2023-03-11 22:36:47 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:51:08.519: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:51:08.519: INFO: pvc-volume-tester-6szft started at 2023-03-11 22:50:58 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:08.519: INFO: 	Container volume-tester ready: true, restart count 0
Mar 11 22:51:08.519: INFO: replace-27976251-8czkq started at 2023-03-11 22:51:00 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:08.519: INFO: 	Container c ready: true, restart count 0
Mar 11 22:51:08.519: INFO: ebs-csi-controller-56487f47bb-qbsdk started at 2023-03-11 22:36:57 +0000 UTC (0+5 container statuses recorded)
Mar 11 22:51:08.519: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:51:08.519: INFO: pod-secrets-558646d9-2fdc-45f1-8b37-5871b0567452 started at 2023-03-11 22:50:01 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:51:08.519: INFO: 	Container creates-volume-test ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container dels-volume-test ready: true, restart count 0
Mar 11 22:51:08.519: INFO: 	Container upds-volume-test ready: true, restart count 0
Mar 11 22:51:08.519: INFO: csi-mockplugin-attacher-0 started at 2023-03-11 22:50:53 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:08.519: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:51:08.519: INFO: webserver-6ff8958fc6-nbmfs started at 2023-03-11 22:51:05 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:08.519: INFO: 	Container httpd ready: true, restart count 0
Mar 11 22:51:09.316: INFO: 
Latency metrics for node i-0ceedc6162dac7c42
Mar 11 22:51:09.316: INFO: 
Logging node info for node i-0ef539cea8af5adb9
Mar 11 22:51:09.532: INFO: Node Info: &Node{ObjectMeta:{i-0ef539cea8af5adb9    b247edc6-7a24-457b-85a5-35f8a510c12c 20344 0 2023-03-11 22:32:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0ef539cea8af5adb9 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0ef539cea8af5adb9"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 22:32:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 22:32:57 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:33:39 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:33:39 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 22:33:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 22:46:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0ef539cea8af5adb9,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4002074624 0} {<nil>} 3908276Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3897217024 0} {<nil>} 3805876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 22:46:16 +0000 UTC,LastTransitionTime:2023-03-11 22:32:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 22:46:16 +0000 UTC,LastTransitionTime:2023-03-11 22:32:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 22:46:16 +0000 UTC,LastTransitionTime:2023-03-11 22:32:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 22:46:16 +0000 UTC,LastTransitionTime:2023-03-11 22:33:39 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.218,},NodeAddress{Type:ExternalIP,Address:54.169.229.126,},NodeAddress{Type:InternalDNS,Address:i-0ef539cea8af5adb9.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ef539cea8af5adb9.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-169-229-126.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2b39ada76eefe97ffcfe201a7e5f77,SystemUUID:ec2b39ad-a76e-efe9-7ffc-fe201a7e5f77,BootID:e1af46ce-9267-4c6e-9738-fdc23cbf769f,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,KubeProxyVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:115123555,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:107914749,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:60946786,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:56731120,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41885191,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40376038,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/external-dns/external-dns@sha256:1529982e95d0c9a1f0543360939108cf8276507fb6639e367a0198d3b25570b0 registry.k8s.io/external-dns/external-dns:v0.13.1],SizeBytes:22414358,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740762,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 22:51:09.533: INFO: 
Logging kubelet events for node i-0ef539cea8af5adb9
Mar 11 22:51:09.753: INFO: 
Logging pods the kubelet thinks is on node i-0ef539cea8af5adb9
Mar 11 22:51:09.995: INFO: kube-controller-manager-i-0ef539cea8af5adb9 started at 2023-03-11 22:30:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:09.995: INFO: 	Container kube-controller-manager ready: true, restart count 4
Mar 11 22:51:09.995: INFO: ebs-csi-node-8ztn8 started at 2023-03-11 22:33:24 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:51:09.995: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:51:09.995: INFO: kops-controller-4b6hj started at 2023-03-11 22:33:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:09.995: INFO: 	Container kops-controller ready: true, restart count 0
Mar 11 22:51:09.995: INFO: cilium-operator-66756dbf95-f9895 started at 2023-03-11 22:33:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:09.995: INFO: 	Container cilium-operator ready: true, restart count 1
Mar 11 22:51:09.995: INFO: cilium-rsmft started at 2023-03-11 22:33:24 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:51:09.995: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Container cilium-agent ready: true, restart count 1
Mar 11 22:51:09.995: INFO: external-dns-5d96c7b864-f8tgk started at 2023-03-11 22:33:39 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:09.995: INFO: 	Container external-dns ready: true, restart count 0
Mar 11 22:51:09.995: INFO: etcd-manager-events-i-0ef539cea8af5adb9 started at 2023-03-11 22:30:24 +0000 UTC (11+1 container statuses recorded)
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 11 22:51:09.995: INFO: etcd-manager-main-i-0ef539cea8af5adb9 started at 2023-03-11 22:30:24 +0000 UTC (11+1 container statuses recorded)
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 11 22:51:09.995: INFO: kube-apiserver-i-0ef539cea8af5adb9 started at 2023-03-11 22:30:24 +0000 UTC (0+2 container statuses recorded)
Mar 11 22:51:09.995: INFO: 	Container healthcheck ready: true, restart count 0
Mar 11 22:51:09.995: INFO: 	Container kube-apiserver ready: true, restart count 3
Mar 11 22:51:09.995: INFO: kube-scheduler-i-0ef539cea8af5adb9 started at 2023-03-11 22:30:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:09.995: INFO: 	Container kube-scheduler ready: true, restart count 0
Mar 11 22:51:09.995: INFO: aws-cloud-controller-manager-jn2dh started at 2023-03-11 22:33:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:51:09.995: INFO: 	Container aws-cloud-controller-manager ready: true, restart count 0
Mar 11 22:51:10.715: INFO: 
Latency metrics for node i-0ef539cea8af5adb9
END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 22:51:10.715 (9.064s)
< Exit [DeferCleanup (Each)] [sig-network] EndpointSlice - dump namespaces | framework.go:209 @ 03/11/23 22:51:10.715 (9.064s)
> Enter [DeferCleanup (Each)] [sig-network] EndpointSlice - tear down framework | framework.go:206 @ 03/11/23 22:51:10.715
STEP: Destroying namespace "endpointslice-8754" for this suite. - test/e2e/framework/framework.go:351 @ 03/11/23 22:51:10.715
< Exit [DeferCleanup (Each)] [sig-network] EndpointSlice - tear down framework | framework.go:206 @ 03/11/23 22:51:10.933 (219ms)
> Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 22:51:10.933
< Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 22:51:10.933 (0s)

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

Kubernetes e2e suite [It] [sig-network] Services should serve endpoints on same port and different protocols 57s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-network\]\sServices\sshould\sserve\sendpoints\son\ssame\sport\sand\sdifferent\sprotocols$'

[FAILED] Timed out after 30.001s.
Expected
    <nil>: nil
not to be nil
In [It] at: test/e2e/network/service.go:3703 @ 03/11/23 22:56:59.946

				> Enter [BeforeEach] [sig-network] Services - set up framework | framework.go:191 @ 03/11/23 22:56:10.963
STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/11/23 22:56:10.963
Mar 11 22:56:10.963: INFO: >>> kubeConfig: /root/.kube/config
STEP: Building a namespace api object, basename services - test/e2e/framework/framework.go:250 @ 03/11/23 22:56:10.964
STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/11/23 22:56:11.614
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/11/23 22:56:12.045
< Exit [BeforeEach] [sig-network] Services - set up framework | framework.go:191 @ 03/11/23 22:56:12.476 (1.513s)
> Enter [BeforeEach] [sig-network] Services - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 22:56:12.476
< Exit [BeforeEach] [sig-network] Services - test/e2e/framework/metrics/init/init.go:33 @ 03/11/23 22:56:12.476 (0s)
> Enter [BeforeEach] [sig-network] Services - test/e2e/network/service.go:764 @ 03/11/23 22:56:12.476
< Exit [BeforeEach] [sig-network] Services - test/e2e/network/service.go:764 @ 03/11/23 22:56:12.476 (0s)
> Enter [It] should serve endpoints on same port and different protocols - test/e2e/network/service.go:3622 @ 03/11/23 22:56:12.476
STEP: creating service multiprotocol-test in namespace services-5254 - test/e2e/network/service.go:3641 @ 03/11/23 22:56:12.476
STEP: creating pod pod1 in namespace services-5254 - test/e2e/network/service.go:3667 @ 03/11/23 22:56:12.702
STEP: Creating pod pod1 in namespace services-5254 - test/e2e/network/service.go:4094 @ 03/11/23 22:56:12.702
STEP: waiting up to 3m0s for service multiprotocol-test in namespace services-5254 to expose endpoints map[pod1:[{tcp-port 0 80 TCP } {udp-port 0 80 UDP }]] - test/e2e/network/service.go:4286 @ 03/11/23 22:56:15.574
Mar 11 22:56:16.447: INFO: successfully validated that service multiprotocol-test in namespace services-5254 exposes endpoints map[pod1:[{tcp-port 0 80 TCP } {udp-port 0 80 UDP }]]
STEP: Checking if the Service forwards traffic to the TCP and UDP port - test/e2e/network/service.go:3671 @ 03/11/23 22:56:16.447
Mar 11 22:56:16.447: INFO: Creating new exec pod
Mar 11 22:56:19.101: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-5254 exec execpodwb4xf -- /bin/sh -x -c echo hostName | nc -v -t -w 2 100.66.51.8 80'
Mar 11 22:56:21.279: INFO: stderr: "+ + echo hostName\nnc -v -t -w 2 100.66.51.8 80\nConnection to 100.66.51.8 80 port [tcp/http] succeeded!\n"
Mar 11 22:56:21.279: INFO: stdout: "HTTP/1.1 400 Bad Request\r\nContent-Type: text/plain; charset=utf-8\r\nConnection: close\r\n\r\n400 Bad Request"
Mar 11 22:56:21.279: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-5254 exec execpodwb4xf -- /bin/sh -x -c echo hostName | nc -v -u -w 2 100.66.51.8 80'
Mar 11 22:56:27.380: INFO: stderr: "+ nc -v -u -w 2 100.66.51.8 80\n+ echo hostName\nConnection to 100.66.51.8 80 port [udp/*] succeeded!\n"
Mar 11 22:56:27.380: INFO: stdout: "pod1"
STEP: Checking if the Service forwards traffic to TCP only - test/e2e/network/service.go:3682 @ 03/11/23 22:56:27.38
Mar 11 22:56:27.817: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-5254 exec execpodwb4xf -- /bin/sh -x -c echo hostName | nc -v -t -w 2 100.66.51.8 80'
Mar 11 22:56:29.944: INFO: stderr: "+ echo hostName\n+ nc -v -t -w 2 100.66.51.8 80\nConnection to 100.66.51.8 80 port [tcp/http] succeeded!\n"
Mar 11 22:56:29.944: INFO: stdout: "HTTP/1.1 400 Bad Request\r\nContent-Type: text/plain; charset=utf-8\r\nConnection: close\r\n\r\n400 Bad Request"
Mar 11 22:56:29.944: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-5254 exec execpodwb4xf -- /bin/sh -x -c echo hostName | nc -v -u -w 2 100.66.51.8 80'
Mar 11 22:56:36.129: INFO: stderr: "+ echo hostName\n+ nc -v -u -w 2 100.66.51.8 80\nConnection to 100.66.51.8 80 port [udp/*] succeeded!\n"
Mar 11 22:56:36.129: INFO: stdout: "pod1"
Mar 11 22:56:41.130: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-5254 exec execpodwb4xf -- /bin/sh -x -c echo hostName | nc -v -u -w 2 100.66.51.8 80'
Mar 11 22:56:47.261: INFO: stderr: "+ nc -v -u -w 2 100.66.51.8 80\n+ echo hostName\nConnection to 100.66.51.8 80 port [udp/*] succeeded!\n"
Mar 11 22:56:47.261: INFO: stdout: "pod1"
Mar 11 22:56:52.262: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/9dbf3b4a-c05b-11ed-9d91-c209e83e3776/kubectl --server=https://api.e2e-e2e-kops-aws-external-dns.test-cncf-aws.k8s.io --kubeconfig=/root/.kube/config --namespace=services-5254 exec execpodwb4xf -- /bin/sh -x -c echo hostName | nc -v -u -w 2 100.66.51.8 80'
Mar 11 22:56:58.396: INFO: stderr: "+ echo hostName\n+ nc -v -u -w 2 100.66.51.8 80\nConnection to 100.66.51.8 80 port [udp/*] succeeded!\n"
Mar 11 22:56:58.396: INFO: stdout: "pod1"
[FAILED] Timed out after 30.001s.
Expected
    <nil>: nil
not to be nil
In [It] at: test/e2e/network/service.go:3703 @ 03/11/23 22:56:59.946
< Exit [It] should serve endpoints on same port and different protocols - test/e2e/network/service.go:3622 @ 03/11/23 22:56:59.946 (47.47s)
> Enter [AfterEach] [sig-network] Services - test/e2e/framework/node/init/init.go:33 @ 03/11/23 22:56:59.946
Mar 11 22:56:59.946: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
< Exit [AfterEach] [sig-network] Services - test/e2e/framework/node/init/init.go:33 @ 03/11/23 22:57:00.163 (218ms)
> Enter [DeferCleanup (Each)] [sig-network] Services - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 22:57:00.163
< Exit [DeferCleanup (Each)] [sig-network] Services - test/e2e/framework/metrics/init/init.go:35 @ 03/11/23 22:57:00.163 (0s)
> Enter [DeferCleanup (Each)] [sig-network] Services - dump namespaces | framework.go:209 @ 03/11/23 22:57:00.163
STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 22:57:00.163
STEP: Collecting events from namespace "services-5254". - test/e2e/framework/debug/dump.go:42 @ 03/11/23 22:57:00.163
STEP: Found 8 events. - test/e2e/framework/debug/dump.go:46 @ 03/11/23 22:57:00.382
Mar 11 22:57:00.382: INFO: At 2023-03-11 22:56:12 +0000 UTC - event for pod1: {default-scheduler } Scheduled: Successfully assigned services-5254/pod1 to i-0ceedc6162dac7c42
Mar 11 22:57:00.383: INFO: At 2023-03-11 22:56:13 +0000 UTC - event for pod1: {kubelet i-0ceedc6162dac7c42} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Mar 11 22:57:00.383: INFO: At 2023-03-11 22:56:13 +0000 UTC - event for pod1: {kubelet i-0ceedc6162dac7c42} Created: Created container agnhost-container
Mar 11 22:57:00.383: INFO: At 2023-03-11 22:56:13 +0000 UTC - event for pod1: {kubelet i-0ceedc6162dac7c42} Started: Started container agnhost-container
Mar 11 22:57:00.383: INFO: At 2023-03-11 22:56:16 +0000 UTC - event for execpodwb4xf: {default-scheduler } Scheduled: Successfully assigned services-5254/execpodwb4xf to i-091642c82a7eddb0e
Mar 11 22:57:00.383: INFO: At 2023-03-11 22:56:17 +0000 UTC - event for execpodwb4xf: {kubelet i-091642c82a7eddb0e} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Mar 11 22:57:00.383: INFO: At 2023-03-11 22:56:17 +0000 UTC - event for execpodwb4xf: {kubelet i-091642c82a7eddb0e} Created: Created container agnhost-container
Mar 11 22:57:00.383: INFO: At 2023-03-11 22:56:17 +0000 UTC - event for execpodwb4xf: {kubelet i-091642c82a7eddb0e} Started: Started container agnhost-container
Mar 11 22:57:00.600: INFO: POD           NODE                 PHASE    GRACE  CONDITIONS
Mar 11 22:57:00.600: INFO: execpodwb4xf  i-091642c82a7eddb0e  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:56:16 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:56:17 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:56:17 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:56:16 +0000 UTC  }]
Mar 11 22:57:00.600: INFO: pod1          i-0ceedc6162dac7c42  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:56:12 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:56:14 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:56:14 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-11 22:56:12 +0000 UTC  }]
Mar 11 22:57:00.600: INFO: 
Mar 11 22:57:01.256: INFO: 
Logging node info for node i-0018d62d281b1a944
Mar 11 22:57:01.472: INFO: Node Info: &Node{ObjectMeta:{i-0018d62d281b1a944    cfdd8574-b9b7-4893-b669-f8e992aa4a0e 47608 0 2023-03-11 22:36:49 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0018d62d281b1a944 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0018d62d281b1a944 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-workload-2621":"i-0018d62d281b1a944","ebs.csi.aws.com":"i-0018d62d281b1a944"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:56:23 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-11 22:56:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0018d62d281b1a944,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027199488 0} {<nil>} 3932812Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922341888 0} {<nil>} 3830412Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 22:56:25 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 22:56:25 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 22:56:25 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 22:56:25 +0000 UTC,LastTransitionTime:2023-03-11 22:37:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.186,},NodeAddress{Type:ExternalIP,Address:54.254.207.62,},NodeAddress{Type:InternalDNS,Address:i-0018d62d281b1a944.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0018d62d281b1a944.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-254-207-62.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25280cf0ec049b9b05ea74b6bddd69,SystemUUID:ec25280c-f0ec-049b-9b05-ea74b6bddd69,BootID:d11e51d8-3003-42dc-be7c-5013a4fdf87b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,KubeProxyVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:60946786,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-01eab817db60c09bd kubernetes.io/csi/ebs.csi.aws.com^vol-09df857382c49901a kubernetes.io/csi/ebs.csi.aws.com^vol-0bf5e12f59b4f39de],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-09df857382c49901a,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0bf5e12f59b4f39de,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-01eab817db60c09bd,DevicePath:,},},Config:nil,},}
Mar 11 22:57:01.472: INFO: 
Logging kubelet events for node i-0018d62d281b1a944
Mar 11 22:57:01.693: INFO: 
Logging pods the kubelet thinks is on node i-0018d62d281b1a944
Mar 11 22:57:01.919: INFO: test-webserver-4989c382-3c93-4caa-80d0-863a811a6ca3 started at 2023-03-11 22:56:10 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:01.919: INFO: 	Container test-webserver ready: false, restart count 0
Mar 11 22:57:01.919: INFO: csi-mockplugin-attacher-0 started at 2023-03-11 22:56:13 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:01.919: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:57:01.919: INFO: csi-mockplugin-0 started at 2023-03-11 22:56:12 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:57:01.919: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:57:01.919: INFO: 	Container driver-registrar ready: true, restart count 0
Mar 11 22:57:01.919: INFO: 	Container mock ready: true, restart count 0
Mar 11 22:57:01.919: INFO: pod-0efffd94-d8ca-4f83-b7ed-1b1908d8b4d1 started at 2023-03-11 22:56:20 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:01.919: INFO: 	Container write-pod ready: true, restart count 0
Mar 11 22:57:01.919: INFO: ebs-csi-node-7jb28 started at 2023-03-11 22:36:50 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:57:01.919: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:57:01.919: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:57:01.919: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:57:01.919: INFO: ss-0 started at 2023-03-11 22:56:54 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:01.919: INFO: 	Container webserver ready: true, restart count 0
Mar 11 22:57:01.919: INFO: pvc-volume-tester-nlg8j started at 2023-03-11 22:56:16 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:01.919: INFO: 	Container volume-tester ready: true, restart count 0
Mar 11 22:57:01.919: INFO: netserver-0 started at 2023-03-11 22:56:46 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:01.919: INFO: 	Container webserver ready: true, restart count 0
Mar 11 22:57:01.919: INFO: cilium-zv9z8 started at 2023-03-11 22:36:50 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:57:01.919: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 22:57:01.919: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 11 22:57:01.919: INFO: ss2-1 started at 2023-03-11 22:56:46 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:01.919: INFO: 	Container webserver ready: false, restart count 0
Mar 11 22:57:01.920: INFO: inline-volume-tester-6649h started at 2023-03-11 22:55:19 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:01.920: INFO: 	Container csi-volume-tester ready: true, restart count 0
Mar 11 22:57:01.920: INFO: csi-hostpathplugin-0 started at 2023-03-11 22:57:00 +0000 UTC (0+7 container statuses recorded)
Mar 11 22:57:01.920: INFO: 	Container csi-attacher ready: false, restart count 0
Mar 11 22:57:01.920: INFO: 	Container csi-provisioner ready: false, restart count 0
Mar 11 22:57:01.920: INFO: 	Container csi-resizer ready: false, restart count 0
Mar 11 22:57:01.920: INFO: 	Container csi-snapshotter ready: false, restart count 0
Mar 11 22:57:01.920: INFO: 	Container hostpath ready: false, restart count 0
Mar 11 22:57:01.920: INFO: 	Container liveness-probe ready: false, restart count 0
Mar 11 22:57:01.920: INFO: 	Container node-driver-registrar ready: false, restart count 0
Mar 11 22:57:02.708: INFO: 
Latency metrics for node i-0018d62d281b1a944
Mar 11 22:57:02.708: INFO: 
Logging node info for node i-091642c82a7eddb0e
Mar 11 22:57:02.925: INFO: Node Info: &Node{ObjectMeta:{i-091642c82a7eddb0e    3c261825-5a9d-400e-bf5d-53c07d5c0bd1 49196 0 2023-03-11 22:36:49 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-091642c82a7eddb0e kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-091642c82a7eddb0e topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-workload-5176":"i-091642c82a7eddb0e","ebs.csi.aws.com":"i-091642c82a7eddb0e"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 22:36:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-03-11 22:56:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 22:57:01 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-091642c82a7eddb0e,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027191296 0} {<nil>} 3932804Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922333696 0} {<nil>} 3830404Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 22:56:44 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 22:56:44 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 22:56:44 +0000 UTC,LastTransitionTime:2023-03-11 22:36:49 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 22:56:44 +0000 UTC,LastTransitionTime:2023-03-11 22:37:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.181,},NodeAddress{Type:ExternalIP,Address:54.179.156.160,},NodeAddress{Type:InternalDNS,Address:i-091642c82a7eddb0e.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-091642c82a7eddb0e.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-179-156-160.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec27d32cf73f1b00b3ac0095c7778d9c,SystemUUID:ec27d32c-f73f-1b00-b3ac-0095c7778d9c,BootID:18c1c9d5-0ef1-4618-a437-4cf7cc76878a,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,KubeProxyVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:60946786,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:21331383,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:18561972,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-07f734f6d8f31a58b],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-workload-5176^0774b383-c060-11ed-b75d-7a125f656cc3,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0663642f3fe3f0ee3,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-07f734f6d8f31a58b,DevicePath:,},},Config:nil,},}
Mar 11 22:57:02.926: INFO: 
Logging kubelet events for node i-091642c82a7eddb0e
Mar 11 22:57:03.147: INFO: 
Logging pods the kubelet thinks is on node i-091642c82a7eddb0e
Mar 11 22:57:03.374: INFO: coredns-745bddcfbb-pj9zs started at 2023-03-11 22:37:11 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container coredns ready: true, restart count 0
Mar 11 22:57:03.374: INFO: hostexec-i-091642c82a7eddb0e-jdjlb started at 2023-03-11 22:57:02 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container agnhost-container ready: false, restart count 0
Mar 11 22:57:03.374: INFO: ebs-csi-node-gjxkc started at 2023-03-11 22:36:49 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:57:03.374: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:57:03.374: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:57:03.374: INFO: pod-disruption-failure-ignore-1-2gjh7 started at 2023-03-11 22:56:38 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container c ready: false, restart count 0
Mar 11 22:57:03.374: INFO: pod-disruption-failure-ignore-2-42pwk started at 2023-03-11 22:56:43 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container c ready: false, restart count 0
Mar 11 22:57:03.374: INFO: pod-disruption-failure-ignore-3-2ncc9 started at 2023-03-11 22:56:47 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container c ready: false, restart count 0
Mar 11 22:57:03.374: INFO: csi-mockplugin-0 started at 2023-03-11 22:56:55 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:57:03.374: INFO: 	Container driver-registrar ready: true, restart count 0
Mar 11 22:57:03.374: INFO: 	Container mock ready: true, restart count 0
Mar 11 22:57:03.374: INFO: csi-mockplugin-attacher-0 started at 2023-03-11 22:56:56 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:57:03.374: INFO: cilium-fsnfv started at 2023-03-11 22:36:49 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 22:57:03.374: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 11 22:57:03.374: INFO: netserver-1 started at 2023-03-11 22:56:46 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container webserver ready: true, restart count 0
Mar 11 22:57:03.374: INFO: busybox-35d5186f-5873-46c4-a1de-295384c70519 started at 2023-03-11 22:55:07 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container busybox ready: true, restart count 0
Mar 11 22:57:03.374: INFO: ss-2 started at 2023-03-11 22:56:25 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container webserver ready: true, restart count 0
Mar 11 22:57:03.374: INFO: pod-4e5e4805-2700-4602-a9c8-2e796c4b9c04 started at 2023-03-11 22:56:59 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container write-pod ready: true, restart count 0
Mar 11 22:57:03.374: INFO: test-grpc-b2359812-17f1-45c0-b067-c7e3ba0eaea5 started at 2023-03-11 22:53:10 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container agnhost ready: true, restart count 0
Mar 11 22:57:03.374: INFO: pod-disruption-failure-ignore-0-hpd4l started at 2023-03-11 22:56:38 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container c ready: true, restart count 0
Mar 11 22:57:03.374: INFO: execpodwb4xf started at 2023-03-11 22:56:16 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 22:57:03.374: INFO: inline-volume-m6vj2 started at 2023-03-11 22:56:18 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container volume-tester ready: false, restart count 0
Mar 11 22:57:03.374: INFO: ss2-0 started at 2023-03-11 22:56:45 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container webserver ready: true, restart count 0
Mar 11 22:57:03.374: INFO: pod-disruption-failure-ignore-0-nh949 started at 2023-03-11 22:56:54 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container c ready: false, restart count 0
Mar 11 22:57:03.374: INFO: pause-pod-1 started at 2023-03-11 22:56:30 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 22:57:03.374: INFO: pvc-volume-tester-nqww6 started at 2023-03-11 22:57:00 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:03.374: INFO: 	Container volume-tester ready: false, restart count 0
Mar 11 22:57:04.161: INFO: 
Latency metrics for node i-091642c82a7eddb0e
Mar 11 22:57:04.161: INFO: 
Logging node info for node i-0a2163d12c183f215
Mar 11 22:57:04.377: INFO: Node Info: &Node{ObjectMeta:{i-0a2163d12c183f215    7102c397-377c-4dde-9536-441cf78d33c9 48947 0 2023-03-11 22:36:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0a2163d12c183f215 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0a2163d12c183f215 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-5609":"i-0a2163d12c183f215","ebs.csi.aws.com":"i-0a2163d12c183f215"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 22:36:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:36:31 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 22:36:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-11 22:36:31 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:36:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:56:54 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-11 22:56:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0a2163d12c183f215,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027191296 0} {<nil>} 3932804Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922333696 0} {<nil>} 3830404Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 22:56:54 +0000 UTC,LastTransitionTime:2023-03-11 22:36:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 22:56:54 +0000 UTC,LastTransitionTime:2023-03-11 22:36:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 22:56:54 +0000 UTC,LastTransitionTime:2023-03-11 22:36:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 22:56:54 +0000 UTC,LastTransitionTime:2023-03-11 22:36:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.63.42,},NodeAddress{Type:ExternalIP,Address:46.137.231.215,},NodeAddress{Type:InternalDNS,Address:i-0a2163d12c183f215.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0a2163d12c183f215.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-46-137-231-215.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec22c6e1f6d754b17f85187d7d62103d,SystemUUID:ec22c6e1-f6d7-54b1-7f85-187d7d62103d,BootID:856c0a60-d0bf-4c9a-b89e-ad079db9a26a,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,KubeProxyVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:60946786,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-5609^fe0cfbee-c05f-11ed-900a-e6aba8f5ffca kubernetes.io/csi/ebs.csi.aws.com^vol-04ad6da6d54e0e5de kubernetes.io/csi/ebs.csi.aws.com^vol-08ba4f82da9590b3a],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-5609^fe0cfbee-c05f-11ed-900a-e6aba8f5ffca,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-04ad6da6d54e0e5de,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-08ba4f82da9590b3a,DevicePath:,},},Config:nil,},}
Mar 11 22:57:04.377: INFO: 
Logging kubelet events for node i-0a2163d12c183f215
Mar 11 22:57:04.597: INFO: 
Logging pods the kubelet thinks is on node i-0a2163d12c183f215
Mar 11 22:57:05.033: INFO: ss-1 started at 2023-03-11 22:56:36 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:05.033: INFO: 	Container webserver ready: true, restart count 0
Mar 11 22:57:05.033: INFO: ebs-csi-node-g7wxp started at 2023-03-11 22:36:32 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:57:05.033: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:57:05.033: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:57:05.033: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:57:05.033: INFO: coredns-745bddcfbb-ldcr7 started at 2023-03-11 22:36:43 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:05.033: INFO: 	Container coredns ready: true, restart count 0
Mar 11 22:57:05.033: INFO: coredns-autoscaler-8495f5575-bj25x started at 2023-03-11 22:36:43 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:05.033: INFO: 	Container autoscaler ready: true, restart count 0
Mar 11 22:57:05.033: INFO: cilium-tsxzj started at 2023-03-11 22:36:32 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:57:05.033: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 22:57:05.033: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 11 22:57:05.033: INFO: ebs-csi-controller-56487f47bb-hm785 started at 2023-03-11 22:36:43 +0000 UTC (0+5 container statuses recorded)
Mar 11 22:57:05.033: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:57:05.033: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:57:05.033: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 22:57:05.033: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:57:05.033: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:57:05.033: INFO: pod-subpath-test-dynamicpv-4cq4 started at 2023-03-11 22:57:02 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:05.033: INFO: 	Container test-container-subpath-dynamicpv-4cq4 ready: false, restart count 0
Mar 11 22:57:05.033: INFO: external-client started at 2023-03-11 22:56:39 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:05.033: INFO: 	Container external-client ready: true, restart count 0
Mar 11 22:57:05.033: INFO: pod-ephm-test-configmap-k94n started at 2023-03-11 22:56:36 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:05.033: INFO: 	Container test-container-subpath-configmap-k94n ready: false, restart count 0
Mar 11 22:57:05.033: INFO: netserver-2 started at 2023-03-11 22:56:47 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:05.033: INFO: 	Container webserver ready: true, restart count 0
Mar 11 22:57:05.033: INFO: csi-hostpathplugin-0 started at 2023-03-11 22:56:42 +0000 UTC (0+7 container statuses recorded)
Mar 11 22:57:05.033: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:57:05.033: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:57:05.033: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 22:57:05.033: INFO: 	Container csi-snapshotter ready: true, restart count 0
Mar 11 22:57:05.033: INFO: 	Container hostpath ready: true, restart count 0
Mar 11 22:57:05.033: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:57:05.033: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:57:05.822: INFO: 
Latency metrics for node i-0a2163d12c183f215
Mar 11 22:57:05.822: INFO: 
Logging node info for node i-0ceedc6162dac7c42
Mar 11 22:57:06.039: INFO: Node Info: &Node{ObjectMeta:{i-0ceedc6162dac7c42    adea5b88-823a-4c79-bfd5-3e46a8aaf8f4 49251 0 2023-03-11 22:36:46 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0ceedc6162dac7c42 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.hostpath.csi/node:i-0ceedc6162dac7c42 topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-attach-5276":"i-0ceedc6162dac7c42","ebs.csi.aws.com":"i-0ceedc6162dac7c42"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-11 22:36:46 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:36:46 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-11 22:36:46 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2023-03-11 22:36:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2023-03-11 22:36:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-03-11 22:57:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-11 22:57:03 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0ceedc6162dac7c42,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027187200 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922329600 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 22:57:02 +0000 UTC,LastTransitionTime:2023-03-11 22:36:46 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 22:57:02 +0000 UTC,LastTransitionTime:2023-03-11 22:36:46 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 22:57:02 +0000 UTC,LastTransitionTime:2023-03-11 22:36:46 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 22:57:02 +0000 UTC,LastTransitionTime:2023-03-11 22:36:57 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.91,},NodeAddress{Type:ExternalIP,Address:52.77.247.227,},NodeAddress{Type:InternalDNS,Address:i-0ceedc6162dac7c42.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ceedc6162dac7c42.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-52-77-247-227.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2956a0e70ff2db5f15382b6e61f0c7,SystemUUID:ec2956a0-e70f-f2db-5f15-382b6e61f0c7,BootID:5df332c2-10b8-4fd4-a70f-4b257ccfb865,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,KubeProxyVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:60946786,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:47953400,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:ee3b525d5b89db99da3b8eb521d9cd90cb6e9ef0fbb651e98bb37be78d36b5b8 registry.k8s.io/sig-storage/csi-provisioner:v3.3.0],SizeBytes:23945474,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2003505,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-attach-5276^0514354b-c060-11ed-9d1f-eebc5b0b1b77],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-attach-5276^0514354b-c060-11ed-9d1f-eebc5b0b1b77,DevicePath:,},},Config:nil,},}
Mar 11 22:57:06.039: INFO: 
Logging kubelet events for node i-0ceedc6162dac7c42
Mar 11 22:57:06.259: INFO: 
Logging pods the kubelet thinks is on node i-0ceedc6162dac7c42
Mar 11 22:57:06.483: INFO: csi-mockplugin-0 started at 2023-03-11 22:56:49 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:57:06.483: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:57:06.483: INFO: 	Container driver-registrar ready: true, restart count 0
Mar 11 22:57:06.483: INFO: 	Container mock ready: true, restart count 0
Mar 11 22:57:06.483: INFO: ebs-csi-controller-56487f47bb-qbsdk started at 2023-03-11 22:36:57 +0000 UTC (0+5 container statuses recorded)
Mar 11 22:57:06.483: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:57:06.483: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 11 22:57:06.483: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 11 22:57:06.483: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:57:06.483: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:57:06.483: INFO: pvc-volume-tester-qz9v5 started at 2023-03-11 22:56:56 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:06.483: INFO: 	Container volume-tester ready: true, restart count 0
Mar 11 22:57:06.483: INFO: csi-mockplugin-0 started at 2023-03-11 22:57:05 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:57:06.483: INFO: 	Container csi-provisioner ready: false, restart count 0
Mar 11 22:57:06.483: INFO: 	Container driver-registrar ready: false, restart count 0
Mar 11 22:57:06.483: INFO: 	Container mock ready: false, restart count 0
Mar 11 22:57:06.483: INFO: exec-volume-test-preprovisionedpv-mfvh started at 2023-03-11 22:57:01 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:06.483: INFO: 	Container exec-container-preprovisionedpv-mfvh ready: false, restart count 0
Mar 11 22:57:06.483: INFO: csi-mockplugin-attacher-0 started at 2023-03-11 22:56:49 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:06.483: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 11 22:57:06.483: INFO: cilium-bhp52 started at 2023-03-11 22:36:47 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:57:06.483: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 22:57:06.483: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 11 22:57:06.483: INFO: hostexec-i-0ceedc6162dac7c42-mgzxc started at 2023-03-11 22:57:02 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:06.483: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 22:57:06.483: INFO: netserver-3 started at 2023-03-11 22:56:47 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:06.483: INFO: 	Container webserver ready: false, restart count 0
Mar 11 22:57:06.483: INFO: hostexec-i-0ceedc6162dac7c42-qk5bq started at 2023-03-11 22:56:48 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:06.483: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 22:57:06.483: INFO: pod-projected-configmaps-c7482ab2-86f3-4691-9953-03ccf05be6ce started at 2023-03-11 22:56:01 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:57:06.483: INFO: 	Container createcm-volume-test ready: true, restart count 0
Mar 11 22:57:06.483: INFO: 	Container delcm-volume-test ready: true, restart count 0
Mar 11 22:57:06.483: INFO: 	Container updcm-volume-test ready: true, restart count 0
Mar 11 22:57:06.483: INFO: ss2-2 started at 2023-03-11 22:57:03 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:06.483: INFO: 	Container webserver ready: true, restart count 0
Mar 11 22:57:06.483: INFO: ebs-csi-node-tk8q8 started at 2023-03-11 22:36:47 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:57:06.483: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:57:06.483: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:57:06.483: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:57:06.483: INFO: csi-mockplugin-attacher-0 started at 2023-03-11 22:57:05 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:06.483: INFO: 	Container csi-attacher ready: false, restart count 0
Mar 11 22:57:06.483: INFO: pod1 started at 2023-03-11 22:56:12 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:06.483: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 11 22:57:07.267: INFO: 
Latency metrics for node i-0ceedc6162dac7c42
Mar 11 22:57:07.267: INFO: 
Logging node info for node i-0ef539cea8af5adb9
Mar 11 22:57:07.484: INFO: Node Info: &Node{ObjectMeta:{i-0ef539cea8af5adb9    b247edc6-7a24-457b-85a5-35f8a510c12c 47723 0 2023-03-11 22:32:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-southeast-1 failure-domain.beta.kubernetes.io/zone:ap-southeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0ef539cea8af5adb9 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.large topology.ebs.csi.aws.com/zone:ap-southeast-1a topology.kubernetes.io/region:ap-southeast-1 topology.kubernetes.io/zone:ap-southeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0ef539cea8af5adb9"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-11 22:32:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-11 22:32:57 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:33:39 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-11 22:33:39 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-11 22:33:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {kubelet Update v1 2023-03-11 22:56:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-southeast-1a/i-0ef539cea8af5adb9,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4002074624 0} {<nil>} 3908276Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3897217024 0} {<nil>} 3805876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-11 22:56:29 +0000 UTC,LastTransitionTime:2023-03-11 22:32:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-11 22:56:29 +0000 UTC,LastTransitionTime:2023-03-11 22:32:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-11 22:56:29 +0000 UTC,LastTransitionTime:2023-03-11 22:32:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-11 22:56:29 +0000 UTC,LastTransitionTime:2023-03-11 22:33:39 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.59.218,},NodeAddress{Type:ExternalIP,Address:54.169.229.126,},NodeAddress{Type:InternalDNS,Address:i-0ef539cea8af5adb9.ap-southeast-1.compute.internal,},NodeAddress{Type:Hostname,Address:i-0ef539cea8af5adb9.ap-southeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-169-229-126.ap-southeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2b39ada76eefe97ffcfe201a7e5f77,SystemUUID:ec2b39ad-a76e-efe9-7ffc-fe201a7e5f77,BootID:e1af46ce-9267-4c6e-9738-fdc23cbf769f,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,KubeProxyVersion:v1.27.0-alpha.3.423+cc3855e0cf6e2d,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:115123555,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:107914749,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:60946786,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.423_cc3855e0cf6e2d],SizeBytes:56731120,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41885191,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40376038,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/external-dns/external-dns@sha256:1529982e95d0c9a1f0543360939108cf8276507fb6639e367a0198d3b25570b0 registry.k8s.io/external-dns/external-dns:v0.13.1],SizeBytes:22414358,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740762,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 11 22:57:07.485: INFO: 
Logging kubelet events for node i-0ef539cea8af5adb9
Mar 11 22:57:07.711: INFO: 
Logging pods the kubelet thinks is on node i-0ef539cea8af5adb9
Mar 11 22:57:07.936: INFO: kube-controller-manager-i-0ef539cea8af5adb9 started at 2023-03-11 22:30:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:07.936: INFO: 	Container kube-controller-manager ready: true, restart count 4
Mar 11 22:57:07.936: INFO: ebs-csi-node-8ztn8 started at 2023-03-11 22:33:24 +0000 UTC (0+3 container statuses recorded)
Mar 11 22:57:07.936: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 11 22:57:07.936: INFO: kops-controller-4b6hj started at 2023-03-11 22:33:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:07.936: INFO: 	Container kops-controller ready: true, restart count 0
Mar 11 22:57:07.936: INFO: etcd-manager-events-i-0ef539cea8af5adb9 started at 2023-03-11 22:30:24 +0000 UTC (11+1 container statuses recorded)
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 11 22:57:07.936: INFO: etcd-manager-main-i-0ef539cea8af5adb9 started at 2023-03-11 22:30:24 +0000 UTC (11+1 container statuses recorded)
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 11 22:57:07.936: INFO: kube-apiserver-i-0ef539cea8af5adb9 started at 2023-03-11 22:30:24 +0000 UTC (0+2 container statuses recorded)
Mar 11 22:57:07.936: INFO: 	Container healthcheck ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Container kube-apiserver ready: true, restart count 3
Mar 11 22:57:07.936: INFO: kube-scheduler-i-0ef539cea8af5adb9 started at 2023-03-11 22:30:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:07.936: INFO: 	Container kube-scheduler ready: true, restart count 0
Mar 11 22:57:07.936: INFO: aws-cloud-controller-manager-jn2dh started at 2023-03-11 22:33:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:07.936: INFO: 	Container aws-cloud-controller-manager ready: true, restart count 0
Mar 11 22:57:07.936: INFO: cilium-operator-66756dbf95-f9895 started at 2023-03-11 22:33:24 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:07.936: INFO: 	Container cilium-operator ready: true, restart count 1
Mar 11 22:57:07.936: INFO: cilium-rsmft started at 2023-03-11 22:33:24 +0000 UTC (1+1 container statuses recorded)
Mar 11 22:57:07.936: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 11 22:57:07.936: INFO: 	Container cilium-agent ready: true, restart count 1
Mar 11 22:57:07.936: INFO: external-dns-5d96c7b864-f8tgk started at 2023-03-11 22:33:39 +0000 UTC (0+1 container statuses recorded)
Mar 11 22:57:07.936: INFO: 	Container external-dns ready: true, restart count 0
Mar 11 22:57:08.653: INFO: 
Latency metrics for node i-0ef539cea8af5adb9
END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/11/23 22:57:08.653 (8.489s)
< Exit [DeferCleanup (Each)] [sig-network] Services - dump namespaces | framework.go:209 @ 03/11/23 22:57:08.653 (8.489s)
> Enter [DeferCleanup (Each)] [sig-network] Services - tear down framework | framework.go:206 @ 03/11/23 22:57:08.653
STEP: Destroying namespace "services-5254" for this suite. - test/e2e/framework/framework.go:351 @ 03/11/23 22:57:08.653
< Exit [DeferCleanup (Each)] [sig-network] Services - tear down framework | framework.go:206 @ 03/11/23 22:57:08.872 (219ms)
> Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 22:57:08.872
< Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/11/23 22:57:08.872 (0s)

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

kubetest2 Test 20m38s

exit status 255
				from junit_runner.xml

Filter through log files | View test history on testgrid

Show 900 Passed Tests

Show 6563 Skipped Tests

Result	FAILURE
Tests	4 failed / 900 succeeded
Started	2023-03-11 22:26
Elapsed	38m24s
Revision	master