e2e-kops-aws-cni-cilium-ipv6 #1636447931236093952 Results

Test Failures

Kubernetes e2e suite [It] [sig-cli] Kubectl client Kubectl patch should add annotations for pods in rc [Conformance] 4.06s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-cli\]\sKubectl\sclient\sKubectl\spatch\sshould\sadd\sannotations\sfor\spods\sin\src\s\s\[Conformance\]$'

[FAILED] error running /home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=kubectl-2849 create -f -:
Command stdout:

stderr:
Unable to connect to the server: stream error: stream ID 1; INTERNAL_ERROR; received from peer

error:
exit status 1
In [It] at: test/e2e/framework/kubectl/builder.go:87 @ 03/16/23 19:35:54.845

				> Enter [BeforeEach] [sig-cli] Kubectl client - set up framework | framework.go:191 @ 03/16/23 19:35:54.059
STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/16/23 19:35:54.059
Mar 16 19:35:54.059: INFO: >>> kubeConfig: /root/.kube/config
STEP: Building a namespace api object, basename kubectl - test/e2e/framework/framework.go:250 @ 03/16/23 19:35:54.06
STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/16/23 19:35:54.265
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/16/23 19:35:54.395
< Exit [BeforeEach] [sig-cli] Kubectl client - set up framework | framework.go:191 @ 03/16/23 19:35:54.522 (464ms)
> Enter [BeforeEach] [sig-cli] Kubectl client - test/e2e/framework/metrics/init/init.go:33 @ 03/16/23 19:35:54.522
< Exit [BeforeEach] [sig-cli] Kubectl client - test/e2e/framework/metrics/init/init.go:33 @ 03/16/23 19:35:54.522 (0s)
> Enter [BeforeEach] [sig-cli] Kubectl client - test/e2e/kubectl/kubectl.go:276 @ 03/16/23 19:35:54.522
< Exit [BeforeEach] [sig-cli] Kubectl client - test/e2e/kubectl/kubectl.go:276 @ 03/16/23 19:35:54.522 (0s)
> Enter [It] should add annotations for pods in rc  [Conformance] - test/e2e/kubectl/kubectl.go:1640 @ 03/16/23 19:35:54.522
STEP: creating Agnhost RC - test/e2e/kubectl/kubectl.go:1642 @ 03/16/23 19:35:54.523
Mar 16 19:35:54.523: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=kubectl-2849 create -f -'
Mar 16 19:35:54.844: INFO: rc: 1
Mar 16 19:35:54.844: INFO: Unexpected error: 
    <exec.CodeExitError>: 
    error running /home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=kubectl-2849 create -f -:
    Command stdout:
    
    stderr:
    Unable to connect to the server: stream error: stream ID 1; INTERNAL_ERROR; received from peer
    
    error:
    exit status 1
    {
        Err: <*errors.errorString | 0xc0012b7770>{
            s: "error running /home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=kubectl-2849 create -f -:\nCommand stdout:\n\nstderr:\nUnable to connect to the server: stream error: stream ID 1; INTERNAL_ERROR; received from peer\n\nerror:\nexit status 1",
        },
        Code: 1,
    }
[FAILED] error running /home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=kubectl-2849 create -f -:
Command stdout:

stderr:
Unable to connect to the server: stream error: stream ID 1; INTERNAL_ERROR; received from peer

error:
exit status 1
In [It] at: test/e2e/framework/kubectl/builder.go:87 @ 03/16/23 19:35:54.845
< Exit [It] should add annotations for pods in rc  [Conformance] - test/e2e/kubectl/kubectl.go:1640 @ 03/16/23 19:35:54.845 (322ms)
> Enter [AfterEach] [sig-cli] Kubectl client - test/e2e/framework/node/init/init.go:33 @ 03/16/23 19:35:54.845
Mar 16 19:35:54.845: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
< Exit [AfterEach] [sig-cli] Kubectl client - test/e2e/framework/node/init/init.go:33 @ 03/16/23 19:35:55.065 (220ms)
> Enter [DeferCleanup (Each)] [sig-cli] Kubectl client - test/e2e/framework/metrics/init/init.go:35 @ 03/16/23 19:35:55.065
< Exit [DeferCleanup (Each)] [sig-cli] Kubectl client - test/e2e/framework/metrics/init/init.go:35 @ 03/16/23 19:35:55.065 (0s)
> Enter [DeferCleanup (Each)] [sig-cli] Kubectl client - dump namespaces | framework.go:209 @ 03/16/23 19:35:55.065
STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/16/23 19:35:55.065
STEP: Collecting events from namespace "kubectl-2849". - test/e2e/framework/debug/dump.go:42 @ 03/16/23 19:35:55.065
STEP: Found 0 events. - test/e2e/framework/debug/dump.go:46 @ 03/16/23 19:35:55.182
Mar 16 19:35:55.274: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Mar 16 19:35:55.274: INFO: 
Mar 16 19:35:55.347: INFO: 
Logging node info for node i-006f135965e189394
Mar 16 19:35:55.419: INFO: Node Info: &Node{ObjectMeta:{i-006f135965e189394    f587fccb-3521-41bc-a111-b85c50902da0 837 0 2023-03-16 19:30:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-006f135965e189394 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.large topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-006f135965e189394"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:2600:1f14:550:7f01:ff27::/80,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-006f135965e189394,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[2600:1f14:550:7f01:ff27::/80],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4002074624 0} {<nil>} 3908276Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3897217024 0} {<nil>} 3805876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-16 19:32:56 +0000 UTC,LastTransitionTime:2023-03-16 19:30:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-16 19:32:56 +0000 UTC,LastTransitionTime:2023-03-16 19:30:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-16 19:32:56 +0000 UTC,LastTransitionTime:2023-03-16 19:30:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-16 19:32:56 +0000 UTC,LastTransitionTime:2023-03-16 19:31:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:2600:1f14:550:7f01:2dc8:a998:6a95:8a0,},NodeAddress{Type:InternalIP,Address:172.20.44.142,},NodeAddress{Type:InternalDNS,Address:i-006f135965e189394.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-006f135965e189394.us-west-2.compute.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec23b983e7b3e74caa3587361d845f96,SystemUUID:ec23b983-e7b3-e74c-aa35-87361d845f96,BootID:f8440a9c-092a-400d-a2ee-e2e49acbe0b8,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.824+c84c8add70281c,KubeProxyVersion:v1.27.0-alpha.3.824+c84c8add70281c,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:115844451,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:108504573,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:67948896,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:57386480,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41884696,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40375541,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740268,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 16 19:35:55.419: INFO: 
Logging kubelet events for node i-006f135965e189394
Mar 16 19:35:55.493: INFO: 
Logging pods the kubelet thinks is on node i-006f135965e189394
Mar 16 19:35:55.774: INFO: dns-controller-68cbff8fc-8fsm9 started at 2023-03-16 19:31:13 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:55.775: INFO: 	Container dns-controller ready: true, restart count 0
Mar 16 19:35:55.775: INFO: kops-controller-zttzd started at 2023-03-16 19:31:13 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:55.775: INFO: 	Container kops-controller ready: true, restart count 0
Mar 16 19:35:55.775: INFO: etcd-manager-events-i-006f135965e189394 started at 2023-03-16 19:28:47 +0000 UTC (11+1 container statuses recorded)
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 16 19:35:55.775: INFO: etcd-manager-main-i-006f135965e189394 started at 2023-03-16 19:28:47 +0000 UTC (11+1 container statuses recorded)
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 16 19:35:55.775: INFO: kube-controller-manager-i-006f135965e189394 started at 2023-03-16 19:28:47 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:55.775: INFO: 	Container kube-controller-manager ready: true, restart count 4
Mar 16 19:35:55.775: INFO: cilium-operator-55dbd764f9-x6vkm started at 2023-03-16 19:31:13 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:55.775: INFO: 	Container cilium-operator ready: true, restart count 0
Mar 16 19:35:55.775: INFO: aws-cloud-controller-manager-2zl7t started at 2023-03-16 19:31:13 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:55.775: INFO: 	Container aws-cloud-controller-manager ready: true, restart count 0
Mar 16 19:35:55.775: INFO: kube-apiserver-i-006f135965e189394 started at 2023-03-16 19:28:47 +0000 UTC (0+2 container statuses recorded)
Mar 16 19:35:55.775: INFO: 	Container healthcheck ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Container kube-apiserver ready: true, restart count 3
Mar 16 19:35:55.775: INFO: kube-scheduler-i-006f135965e189394 started at 2023-03-16 19:28:47 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:55.775: INFO: 	Container kube-scheduler ready: true, restart count 0
Mar 16 19:35:55.775: INFO: ebs-csi-node-2dx52 started at 2023-03-16 19:31:13 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:35:55.775: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:35:55.775: INFO: cilium-d786k started at 2023-03-16 19:31:13 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:35:55.775: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 16 19:35:55.775: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 16 19:35:56.070: INFO: 
Latency metrics for node i-006f135965e189394
Mar 16 19:35:56.070: INFO: 
Logging node info for node i-02c2ffb2c76c516de
Mar 16 19:35:56.134: INFO: Node Info: &Node{ObjectMeta:{i-02c2ffb2c76c516de    a5a98d76-d961-492a-a0ae-b628a2af0a9b 978 0 2023-03-16 19:32:43 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-02c2ffb2c76c516de kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02c2ffb2c76c516de"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-16 19:32:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-16 19:32:43 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-16 19:32:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"2600:1f14:550:7f00:d8a4::/80\"":{}}}} } {kube-controller-manager Update v1 2023-03-16 19:32:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {kubelet Update v1 2023-03-16 19:32:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-03-16 19:33:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:2600:1f14:550:7f00:d8a4::/80,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-02c2ffb2c76c516de,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[2600:1f14:550:7f00:d8a4::/80],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027191296 0} {<nil>} 3932804Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922333696 0} {<nil>} 3830404Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-16 19:33:14 +0000 UTC,LastTransitionTime:2023-03-16 19:32:43 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-16 19:33:14 +0000 UTC,LastTransitionTime:2023-03-16 19:32:43 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-16 19:33:14 +0000 UTC,LastTransitionTime:2023-03-16 19:32:43 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-16 19:33:14 +0000 UTC,LastTransitionTime:2023-03-16 19:32:52 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:2600:1f14:550:7f00:3579:1da1:4725:2f37,},NodeAddress{Type:InternalDNS,Address:i-02c2ffb2c76c516de.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-02c2ffb2c76c516de.us-west-2.compute.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec28d06d73efdbb6fe07a02217c97a2a,SystemUUID:ec28d06d-73ef-dbb6-fe07-a02217c97a2a,BootID:73530b10-f85c-4a38-9eb8-a746f140455a,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.824+c84c8add70281c,KubeProxyVersion:v1.27.0-alpha.3.824+c84c8add70281c,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:67948896,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 16 19:35:56.135: INFO: 
Logging kubelet events for node i-02c2ffb2c76c516de
Mar 16 19:35:56.205: INFO: 
Logging pods the kubelet thinks is on node i-02c2ffb2c76c516de
Mar 16 19:35:56.297: INFO: ebs-csi-controller-74c4b4f9d9-6rpx4 started at 2023-03-16 19:32:54 +0000 UTC (0+5 container statuses recorded)
Mar 16 19:35:56.297: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:35:56.297: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:35:56.297: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 16 19:35:56.297: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:35:56.297: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:35:56.297: INFO: sample-webhook-deployment-7497495989-dpjtv started at 2023-03-16 19:35:54 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:56.297: INFO: 	Container sample-webhook ready: false, restart count 0
Mar 16 19:35:56.297: INFO: startup-0967ea7d-27d4-4ffe-95c9-44cb227c0648 started at 2023-03-16 19:35:54 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:56.297: INFO: 	Container busybox ready: false, restart count 0
Mar 16 19:35:56.297: INFO: dns-test-b027da93-9bb5-4109-b052-9ba798e2b63b started at 2023-03-16 19:35:54 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:35:56.297: INFO: 	Container jessie-querier ready: false, restart count 0
Mar 16 19:35:56.297: INFO: 	Container querier ready: false, restart count 0
Mar 16 19:35:56.297: INFO: 	Container webserver ready: false, restart count 0
Mar 16 19:35:56.297: INFO: hostexec-i-02c2ffb2c76c516de-shlfw started at 2023-03-16 19:35:55 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:56.297: INFO: 	Container agnhost-container ready: false, restart count 0
Mar 16 19:35:56.297: INFO: cilium-rgmxw started at 2023-03-16 19:32:44 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:35:56.297: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 16 19:35:56.298: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 16 19:35:56.298: INFO: ebs-csi-node-dklhf started at 2023-03-16 19:32:44 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:35:56.298: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:35:56.298: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:35:56.298: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:35:56.564: INFO: 
Latency metrics for node i-02c2ffb2c76c516de
Mar 16 19:35:56.564: INFO: 
Logging node info for node i-02ed4b9ad83b92b97
Mar 16 19:35:56.630: INFO: Node Info: &Node{ObjectMeta:{i-02ed4b9ad83b92b97    d9759ff2-9dc9-4ed2-9309-62f8c3e8e9f7 1005 0 2023-03-16 19:32:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-02ed4b9ad83b92b97 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02ed4b9ad83b92b97"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-03-16 19:32:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {kubelet Update v1 2023-03-16 19:32:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-16 19:32:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-16 19:32:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-16 19:32:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"2600:1f14:550:7f00:ae09::/80\"":{}}}} } {kubelet Update v1 2023-03-16 19:33:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:2600:1f14:550:7f00:ae09::/80,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-02ed4b9ad83b92b97,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[2600:1f14:550:7f00:ae09::/80],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027191296 0} {<nil>} 3932804Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922333696 0} {<nil>} 3830404Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-16 19:33:18 +0000 UTC,LastTransitionTime:2023-03-16 19:32:47 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-16 19:33:18 +0000 UTC,LastTransitionTime:2023-03-16 19:32:47 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-16 19:33:18 +0000 UTC,LastTransitionTime:2023-03-16 19:32:47 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-16 19:33:18 +0000 UTC,LastTransitionTime:2023-03-16 19:32:58 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:2600:1f14:550:7f00:1812:ef4a:9c80:6345,},NodeAddress{Type:InternalDNS,Address:i-02ed4b9ad83b92b97.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-02ed4b9ad83b92b97.us-west-2.compute.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2036eeaef2debf99dbb8a752e1b8c1,SystemUUID:ec2036ee-aef2-debf-99db-b8a752e1b8c1,BootID:6eee1f7e-fba6-494b-9d76-2fbfdff3eab5,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.824+c84c8add70281c,KubeProxyVersion:v1.27.0-alpha.3.824+c84c8add70281c,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:67948896,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 16 19:35:56.630: INFO: 
Logging kubelet events for node i-02ed4b9ad83b92b97
Mar 16 19:35:56.704: INFO: 
Logging pods the kubelet thinks is on node i-02ed4b9ad83b92b97
Mar 16 19:35:56.856: INFO: coredns-745bddcfbb-hscd7 started at 2023-03-16 19:33:07 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:56.856: INFO: 	Container coredns ready: true, restart count 0
Mar 16 19:35:56.856: INFO: pod-csi-inline-volumes started at 2023-03-16 19:35:54 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:56.856: INFO: 	Container pod-csi-inline-volumes ready: false, restart count 0
Mar 16 19:35:56.856: INFO: metadata-volume-139aad12-7026-4da4-ad44-32c330b5b753 started at 2023-03-16 19:35:54 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:56.856: INFO: 	Container client-container ready: false, restart count 0
Mar 16 19:35:56.856: INFO: pod-40f7bcdf-9e16-4ddc-8d83-643864637e60 started at 2023-03-16 19:35:54 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:56.856: INFO: 	Container test-container ready: false, restart count 0
Mar 16 19:35:56.856: INFO: termination-message-containerdbf0b756-3e01-4474-a658-aa7d5277d98a started at 2023-03-16 19:35:54 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:56.856: INFO: 	Container termination-message-container ready: false, restart count 0
Mar 16 19:35:56.856: INFO: test-new-deployment-67bd4bf6dc-cxrq8 started at 2023-03-16 19:35:54 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:56.856: INFO: 	Container httpd ready: false, restart count 0
Mar 16 19:35:56.856: INFO: pod-terminate-status-2-0 started at 2023-03-16 19:35:55 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:35:56.856: INFO: 	Init container fail ready: false, restart count 0
Mar 16 19:35:56.856: INFO: 	Container blocked ready: false, restart count 0
Mar 16 19:35:56.856: INFO: ebs-csi-node-74c5w started at 2023-03-16 19:32:48 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:35:56.856: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:35:56.856: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:35:56.856: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:35:56.856: INFO: pod-terminate-status-1-0 started at 2023-03-16 19:35:55 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:35:56.856: INFO: 	Init container fail ready: false, restart count 0
Mar 16 19:35:56.856: INFO: 	Container blocked ready: false, restart count 0
Mar 16 19:35:56.856: INFO: pod-handle-http-request started at 2023-03-16 19:35:54 +0000 UTC (0+2 container statuses recorded)
Mar 16 19:35:56.856: INFO: 	Container container-handle-http-request ready: false, restart count 0
Mar 16 19:35:56.856: INFO: 	Container container-handle-https-request ready: false, restart count 0
Mar 16 19:35:56.856: INFO: client-containers-72fa780f-924e-4248-b4ca-5494e0a9ea11 started at 2023-03-16 19:35:55 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:56.856: INFO: 	Container agnhost-container ready: false, restart count 0
Mar 16 19:35:56.856: INFO: cilium-htdgf started at 2023-03-16 19:32:48 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:35:56.856: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 16 19:35:56.856: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 16 19:35:57.093: INFO: 
Latency metrics for node i-02ed4b9ad83b92b97
Mar 16 19:35:57.093: INFO: 
Logging node info for node i-07b8283b27a926ea4
Mar 16 19:35:57.158: INFO: Node Info: &Node{ObjectMeta:{i-07b8283b27a926ea4    2926fed8-06e5-47f1-80a5-351cf76c4d38 1099 0 2023-03-16 19:32:41 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-07b8283b27a926ea4 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-07b8283b27a926ea4"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-16 19:32:41 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-16 19:32:41 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-16 19:32:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kops-controller Update v1 2023-03-16 19:32:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"2600:1f14:550:7f00:182a::/80\"":{}}}} } {kube-controller-manager Update v1 2023-03-16 19:32:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {kubelet Update v1 2023-03-16 19:33:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:2600:1f14:550:7f00:182a::/80,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-07b8283b27a926ea4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[2600:1f14:550:7f00:182a::/80],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027187200 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922329600 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-16 19:33:42 +0000 UTC,LastTransitionTime:2023-03-16 19:32:41 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-16 19:33:42 +0000 UTC,LastTransitionTime:2023-03-16 19:32:41 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-16 19:33:42 +0000 UTC,LastTransitionTime:2023-03-16 19:32:41 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-16 19:33:42 +0000 UTC,LastTransitionTime:2023-03-16 19:32:51 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:2600:1f14:550:7f00:58e6:44e0:8dbc:9afe,},NodeAddress{Type:InternalDNS,Address:i-07b8283b27a926ea4.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-07b8283b27a926ea4.us-west-2.compute.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2014203de088435d46874c53715c1c,SystemUUID:ec201420-3de0-8843-5d46-874c53715c1c,BootID:9a378fc2-0572-4834-8107-b622fa5e0388,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.824+c84c8add70281c,KubeProxyVersion:v1.27.0-alpha.3.824+c84c8add70281c,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:67948896,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 16 19:35:57.158: INFO: 
Logging kubelet events for node i-07b8283b27a926ea4
Mar 16 19:35:57.226: INFO: 
Logging pods the kubelet thinks is on node i-07b8283b27a926ea4
Mar 16 19:35:57.315: INFO: cilium-fs2kv started at 2023-03-16 19:32:42 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:35:57.316: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 16 19:35:57.316: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 16 19:35:57.316: INFO: ebs-csi-controller-74c4b4f9d9-6l4cr started at 2023-03-16 19:32:51 +0000 UTC (0+5 container statuses recorded)
Mar 16 19:35:57.316: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:35:57.316: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:35:57.316: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 16 19:35:57.316: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:35:57.316: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:35:57.316: INFO: busybox-privileged-true-b60b9fd2-b156-4465-afa5-bbafc81afe8a started at 2023-03-16 19:35:55 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:57.316: INFO: 	Container busybox-privileged-true-b60b9fd2-b156-4465-afa5-bbafc81afe8a ready: false, restart count 0
Mar 16 19:35:57.316: INFO: pod-4591a298-17c0-4454-86f9-6a915fa0babb started at 2023-03-16 19:35:54 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:57.316: INFO: 	Container test-container ready: false, restart count 0
Mar 16 19:35:57.316: INFO: ss2-0 started at 2023-03-16 19:35:55 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:57.316: INFO: 	Container webserver ready: false, restart count 0
Mar 16 19:35:57.316: INFO: pod-terminate-status-0-0 started at 2023-03-16 19:35:55 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:35:57.316: INFO: 	Init container fail ready: false, restart count 0
Mar 16 19:35:57.316: INFO: 	Container blocked ready: false, restart count 0
Mar 16 19:35:57.316: INFO: agnhost-host-aliasesf255afec-b1e0-40d6-ae18-e2571e0aa2f4 started at 2023-03-16 19:35:55 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:57.316: INFO: 	Container agnhost-container ready: false, restart count 0
Mar 16 19:35:57.316: INFO: ebs-csi-node-vqqvw started at 2023-03-16 19:32:42 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:35:57.316: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:35:57.316: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:35:57.316: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:35:57.316: INFO: coredns-745bddcfbb-bmht8 started at 2023-03-16 19:32:51 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:57.316: INFO: 	Container coredns ready: true, restart count 0
Mar 16 19:35:57.316: INFO: coredns-autoscaler-8495f5575-8hwhz started at 2023-03-16 19:32:51 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:57.316: INFO: 	Container autoscaler ready: true, restart count 0
Mar 16 19:35:57.594: INFO: 
Latency metrics for node i-07b8283b27a926ea4
Mar 16 19:35:57.594: INFO: 
Logging node info for node i-0b5975e41089e600d
Mar 16 19:35:57.659: INFO: Node Info: &Node{ObjectMeta:{i-0b5975e41089e600d    13d7c5c9-ada0-4e0c-a9b4-6c3b0e64b4c3 1030 0 2023-03-16 19:32:53 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0b5975e41089e600d kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0b5975e41089e600d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-16 19:32:53 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-16 19:32:53 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-16 19:32:53 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"2600:1f14:550:7f00:f985::/80\"":{}}}} } {kube-controller-manager Update v1 2023-03-16 19:32:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {kubelet Update v1 2023-03-16 19:32:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-03-16 19:33:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:2600:1f14:550:7f00:f985::/80,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0b5975e41089e600d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[2600:1f14:550:7f00:f985::/80],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027187200 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922329600 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-16 19:33:24 +0000 UTC,LastTransitionTime:2023-03-16 19:32:53 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-16 19:33:24 +0000 UTC,LastTransitionTime:2023-03-16 19:32:53 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-16 19:33:24 +0000 UTC,LastTransitionTime:2023-03-16 19:32:53 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-16 19:33:24 +0000 UTC,LastTransitionTime:2023-03-16 19:33:02 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:2600:1f14:550:7f00:c0aa:5f30:7328:8e3f,},NodeAddress{Type:InternalDNS,Address:i-0b5975e41089e600d.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0b5975e41089e600d.us-west-2.compute.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2173701442f95a91d69a4b0877f610,SystemUUID:ec217370-1442-f95a-91d6-9a4b0877f610,BootID:7b20997a-d9ac-4e40-b5c7-ba313b4b084b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.824+c84c8add70281c,KubeProxyVersion:v1.27.0-alpha.3.824+c84c8add70281c,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:67948896,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 16 19:35:57.660: INFO: 
Logging kubelet events for node i-0b5975e41089e600d
Mar 16 19:35:57.727: INFO: 
Logging pods the kubelet thinks is on node i-0b5975e41089e600d
Mar 16 19:35:57.814: INFO: cilium-9vmm6 started at 2023-03-16 19:32:54 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:35:57.814: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 16 19:35:57.814: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 16 19:35:57.814: INFO: ebs-csi-node-rkwb9 started at 2023-03-16 19:32:54 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:35:57.814: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:35:57.814: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:35:57.814: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:35:57.814: INFO: pfpod started at 2023-03-16 19:35:54 +0000 UTC (0+2 container statuses recorded)
Mar 16 19:35:57.814: INFO: 	Container portforwardtester ready: false, restart count 0
Mar 16 19:35:57.814: INFO: 	Container readiness ready: false, restart count 0
Mar 16 19:35:57.814: INFO: downwardapi-volume-f7aef5f0-c7fc-424a-9efe-6c68719ad793 started at 2023-03-16 19:35:54 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:57.814: INFO: 	Container client-container ready: false, restart count 0
Mar 16 19:35:57.814: INFO: downward-api-d42572d4-89e1-4d77-856c-2e607d5564f7 started at 2023-03-16 19:35:54 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:57.814: INFO: 	Container dapi-container ready: false, restart count 0
Mar 16 19:35:57.814: INFO: downwardapi-volume-1448d259-d1b3-45c3-a3f1-50f47fbe2ff2 started at 2023-03-16 19:35:54 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:57.814: INFO: 	Container client-container ready: false, restart count 0
Mar 16 19:35:57.814: INFO: pod-disruption-failure-ignore-0-97mrj started at 2023-03-16 19:35:55 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:57.814: INFO: 	Container c ready: false, restart count 0
Mar 16 19:35:57.814: INFO: pod-disruption-failure-ignore-1-j5s4q started at 2023-03-16 19:35:55 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:35:57.814: INFO: 	Container c ready: false, restart count 0
Mar 16 19:35:58.053: INFO: 
Latency metrics for node i-0b5975e41089e600d
END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/16/23 19:35:58.053 (2.988s)
< Exit [DeferCleanup (Each)] [sig-cli] Kubectl client - dump namespaces | framework.go:209 @ 03/16/23 19:35:58.053 (2.988s)
> Enter [DeferCleanup (Each)] [sig-cli] Kubectl client - tear down framework | framework.go:206 @ 03/16/23 19:35:58.053
STEP: Destroying namespace "kubectl-2849" for this suite. - test/e2e/framework/framework.go:351 @ 03/16/23 19:35:58.053
< Exit [DeferCleanup (Each)] [sig-cli] Kubectl client - tear down framework | framework.go:206 @ 03/16/23 19:35:58.12 (67ms)
> Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/16/23 19:35:58.12
< Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/16/23 19:35:58.12 (0s)

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

Kubernetes e2e suite [It] [sig-network] EndpointSlice should support a Service with multiple ports specified in multiple EndpointSlices 2m20s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-network\]\sEndpointSlice\sshould\ssupport\sa\sService\swith\smultiple\sports\sspecified\sin\smultiple\sEndpointSlices$'

[FAILED] error running /home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=endpointslice-8251 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 [fd00:5e4f:ce::574f]:81/hostname:
Command stdout:

stderr:
+ curl -q -s --connect-timeout 30 '[fd00:5e4f:ce::574f]:81/hostname'
command terminated with exit code 28

error:
exit status 28
In [It] at: test/e2e/network/util.go:180 @ 03/16/23 19:43:43.116

				> Enter [BeforeEach] [sig-network] EndpointSlice - set up framework | framework.go:191 @ 03/16/23 19:41:25.972
STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/16/23 19:41:25.972
Mar 16 19:41:25.972: INFO: >>> kubeConfig: /root/.kube/config
STEP: Building a namespace api object, basename endpointslice - test/e2e/framework/framework.go:250 @ 03/16/23 19:41:25.973
STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/16/23 19:41:26.174
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/16/23 19:41:26.308
< Exit [BeforeEach] [sig-network] EndpointSlice - set up framework | framework.go:191 @ 03/16/23 19:41:26.445 (473ms)
> Enter [BeforeEach] [sig-network] EndpointSlice - test/e2e/framework/metrics/init/init.go:33 @ 03/16/23 19:41:26.445
< Exit [BeforeEach] [sig-network] EndpointSlice - test/e2e/framework/metrics/init/init.go:33 @ 03/16/23 19:41:26.445 (0s)
> Enter [BeforeEach] [sig-network] EndpointSlice - test/e2e/network/endpointslice.go:54 @ 03/16/23 19:41:26.445
< Exit [BeforeEach] [sig-network] EndpointSlice - test/e2e/network/endpointslice.go:54 @ 03/16/23 19:41:26.445 (0s)
> Enter [It] should support a Service with multiple ports specified in multiple EndpointSlices - test/e2e/network/endpointslice.go:530 @ 03/16/23 19:41:26.445
STEP: creating - test/e2e/network/endpointslice.go:597 @ 03/16/23 19:41:28.802
STEP: Creating a pause pods that will try to connect to the webserver - test/e2e/network/endpointslice.go:618 @ 03/16/23 19:41:28.935
Mar 16 19:41:31.207: INFO: Waiting up to 2m0s to get response from [fd00:5e4f:ce::574f]:80
Mar 16 19:41:31.207: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=endpointslice-8251 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 [fd00:5e4f:ce::574f]:80/hostname'
Mar 16 19:41:31.987: INFO: stderr: "+ curl -q -s --connect-timeout 30 '[fd00:5e4f:ce::574f]:80/hostname'\n"
Mar 16 19:41:31.987: INFO: stdout: "pod-handle-http-request"
Mar 16 19:41:31.987: INFO: Waiting up to 2m0s to get response from [fd00:5e4f:ce::574f]:81
Mar 16 19:41:31.987: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=endpointslice-8251 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 [fd00:5e4f:ce::574f]:81/hostname'
Mar 16 19:42:02.781: INFO: rc: 28
Mar 16 19:42:02.781: INFO: got err: error running /home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=endpointslice-8251 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 [fd00:5e4f:ce::574f]:81/hostname:
Command stdout:

stderr:
+ curl -q -s --connect-timeout 30 '[fd00:5e4f:ce::574f]:81/hostname'
command terminated with exit code 28

error:
exit status 28, retry until timeout
Mar 16 19:42:04.782: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=endpointslice-8251 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 [fd00:5e4f:ce::574f]:81/hostname'
Mar 16 19:42:35.552: INFO: rc: 28
Mar 16 19:42:35.552: INFO: got err: error running /home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=endpointslice-8251 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 [fd00:5e4f:ce::574f]:81/hostname:
Command stdout:

stderr:
+ curl -q -s --connect-timeout 30 '[fd00:5e4f:ce::574f]:81/hostname'
command terminated with exit code 28

error:
exit status 28, retry until timeout
Mar 16 19:42:37.553: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=endpointslice-8251 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 [fd00:5e4f:ce::574f]:81/hostname'
Mar 16 19:43:08.328: INFO: rc: 28
Mar 16 19:43:08.328: INFO: got err: error running /home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=endpointslice-8251 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 [fd00:5e4f:ce::574f]:81/hostname:
Command stdout:

stderr:
+ curl -q -s --connect-timeout 30 '[fd00:5e4f:ce::574f]:81/hostname'
command terminated with exit code 28

error:
exit status 28, retry until timeout
Mar 16 19:43:10.329: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=endpointslice-8251 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 [fd00:5e4f:ce::574f]:81/hostname'
Mar 16 19:43:41.115: INFO: rc: 28
Mar 16 19:43:41.115: INFO: got err: error running /home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=endpointslice-8251 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 [fd00:5e4f:ce::574f]:81/hostname:
Command stdout:

stderr:
+ curl -q -s --connect-timeout 30 '[fd00:5e4f:ce::574f]:81/hostname'
command terminated with exit code 28

error:
exit status 28, retry until timeout
Mar 16 19:43:43.116: INFO: Unexpected error: 
    <exec.CodeExitError>: 
    error running /home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=endpointslice-8251 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 [fd00:5e4f:ce::574f]:81/hostname:
    Command stdout:
    
    stderr:
    + curl -q -s --connect-timeout 30 '[fd00:5e4f:ce::574f]:81/hostname'
    command terminated with exit code 28
    
    error:
    exit status 28
    {
        Err: <*errors.errorString | 0xc001702010>{
            s: "error running /home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=endpointslice-8251 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 [fd00:5e4f:ce::574f]:81/hostname:\nCommand stdout:\n\nstderr:\n+ curl -q -s --connect-timeout 30 '[fd00:5e4f:ce::574f]:81/hostname'\ncommand terminated with exit code 28\n\nerror:\nexit status 28",
        },
        Code: 28,
    }
[FAILED] error running /home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=endpointslice-8251 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --connect-timeout 30 [fd00:5e4f:ce::574f]:81/hostname:
Command stdout:

stderr:
+ curl -q -s --connect-timeout 30 '[fd00:5e4f:ce::574f]:81/hostname'
command terminated with exit code 28

error:
exit status 28
In [It] at: test/e2e/network/util.go:180 @ 03/16/23 19:43:43.116
< Exit [It] should support a Service with multiple ports specified in multiple EndpointSlices - test/e2e/network/endpointslice.go:530 @ 03/16/23 19:43:43.116 (2m16.672s)
> Enter [AfterEach] [sig-network] EndpointSlice - test/e2e/framework/node/init/init.go:33 @ 03/16/23 19:43:43.116
Mar 16 19:43:43.116: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
< Exit [AfterEach] [sig-network] EndpointSlice - test/e2e/framework/node/init/init.go:33 @ 03/16/23 19:43:43.183 (66ms)
> Enter [DeferCleanup (Each)] [sig-network] EndpointSlice - test/e2e/framework/metrics/init/init.go:35 @ 03/16/23 19:43:43.183
< Exit [DeferCleanup (Each)] [sig-network] EndpointSlice - test/e2e/framework/metrics/init/init.go:35 @ 03/16/23 19:43:43.183 (0s)
> Enter [DeferCleanup (Each)] [sig-network] EndpointSlice - dump namespaces | framework.go:209 @ 03/16/23 19:43:43.183
STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/16/23 19:43:43.183
STEP: Collecting events from namespace "endpointslice-8251". - test/e2e/framework/debug/dump.go:42 @ 03/16/23 19:43:43.183
STEP: Found 11 events. - test/e2e/framework/debug/dump.go:46 @ 03/16/23 19:43:43.249
Mar 16 19:43:43.249: INFO: At 2023-03-16 19:41:26 +0000 UTC - event for pod-handle-http-request: {default-scheduler } Scheduled: Successfully assigned endpointslice-8251/pod-handle-http-request to i-0b5975e41089e600d
Mar 16 19:43:43.249: INFO: At 2023-03-16 19:41:27 +0000 UTC - event for pod-handle-http-request: {kubelet i-0b5975e41089e600d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Mar 16 19:43:43.249: INFO: At 2023-03-16 19:41:27 +0000 UTC - event for pod-handle-http-request: {kubelet i-0b5975e41089e600d} Created: Created container container-handle-8090-request
Mar 16 19:43:43.249: INFO: At 2023-03-16 19:41:27 +0000 UTC - event for pod-handle-http-request: {kubelet i-0b5975e41089e600d} Started: Started container container-handle-8090-request
Mar 16 19:43:43.249: INFO: At 2023-03-16 19:41:27 +0000 UTC - event for pod-handle-http-request: {kubelet i-0b5975e41089e600d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Mar 16 19:43:43.249: INFO: At 2023-03-16 19:41:27 +0000 UTC - event for pod-handle-http-request: {kubelet i-0b5975e41089e600d} Created: Created container container-handle-9090-request
Mar 16 19:43:43.249: INFO: At 2023-03-16 19:41:27 +0000 UTC - event for pod-handle-http-request: {kubelet i-0b5975e41089e600d} Started: Started container container-handle-9090-request
Mar 16 19:43:43.249: INFO: At 2023-03-16 19:41:28 +0000 UTC - event for pause-pod-0: {default-scheduler } Scheduled: Successfully assigned endpointslice-8251/pause-pod-0 to i-0b5975e41089e600d
Mar 16 19:43:43.249: INFO: At 2023-03-16 19:41:29 +0000 UTC - event for pause-pod-0: {kubelet i-0b5975e41089e600d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Mar 16 19:43:43.249: INFO: At 2023-03-16 19:41:29 +0000 UTC - event for pause-pod-0: {kubelet i-0b5975e41089e600d} Created: Created container agnhost-container
Mar 16 19:43:43.249: INFO: At 2023-03-16 19:41:29 +0000 UTC - event for pause-pod-0: {kubelet i-0b5975e41089e600d} Started: Started container agnhost-container
Mar 16 19:43:43.314: INFO: POD                      NODE                 PHASE    GRACE  CONDITIONS
Mar 16 19:43:43.314: INFO: pause-pod-0              i-0b5975e41089e600d  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:41:28 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:41:30 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:41:30 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:41:28 +0000 UTC  }]
Mar 16 19:43:43.314: INFO: pod-handle-http-request  i-0b5975e41089e600d  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:41:26 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:41:28 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:41:28 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:41:26 +0000 UTC  }]
Mar 16 19:43:43.314: INFO: 
Mar 16 19:43:43.594: INFO: 
Logging node info for node i-006f135965e189394
Mar 16 19:43:43.659: INFO: Node Info: &Node{ObjectMeta:{i-006f135965e189394    f587fccb-3521-41bc-a111-b85c50902da0 26928 0 2023-03-16 19:30:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-006f135965e189394 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.large topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-006f135965e189394"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:2600:1f14:550:7f01:ff27::/80,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-006f135965e189394,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[2600:1f14:550:7f01:ff27::/80],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4002074624 0} {<nil>} 3908276Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3897217024 0} {<nil>} 3805876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-16 19:43:08 +0000 UTC,LastTransitionTime:2023-03-16 19:30:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-16 19:43:08 +0000 UTC,LastTransitionTime:2023-03-16 19:30:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-16 19:43:08 +0000 UTC,LastTransitionTime:2023-03-16 19:30:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-16 19:43:08 +0000 UTC,LastTransitionTime:2023-03-16 19:31:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:2600:1f14:550:7f01:2dc8:a998:6a95:8a0,},NodeAddress{Type:InternalIP,Address:172.20.44.142,},NodeAddress{Type:InternalDNS,Address:i-006f135965e189394.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-006f135965e189394.us-west-2.compute.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec23b983e7b3e74caa3587361d845f96,SystemUUID:ec23b983-e7b3-e74c-aa35-87361d845f96,BootID:f8440a9c-092a-400d-a2ee-e2e49acbe0b8,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.824+c84c8add70281c,KubeProxyVersion:v1.27.0-alpha.3.824+c84c8add70281c,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:115844451,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:108504573,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:67948896,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:57386480,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41884696,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40375541,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740268,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 16 19:43:43.659: INFO: 
Logging kubelet events for node i-006f135965e189394
Mar 16 19:43:43.728: INFO: 
Logging pods the kubelet thinks is on node i-006f135965e189394
Mar 16 19:43:43.867: INFO: cilium-d786k started at 2023-03-16 19:31:13 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:43:43.867: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 16 19:43:43.867: INFO: aws-cloud-controller-manager-2zl7t started at 2023-03-16 19:31:13 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:43.867: INFO: 	Container aws-cloud-controller-manager ready: true, restart count 0
Mar 16 19:43:43.867: INFO: kube-apiserver-i-006f135965e189394 started at 2023-03-16 19:28:47 +0000 UTC (0+2 container statuses recorded)
Mar 16 19:43:43.867: INFO: 	Container healthcheck ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Container kube-apiserver ready: true, restart count 3
Mar 16 19:43:43.867: INFO: kube-scheduler-i-006f135965e189394 started at 2023-03-16 19:28:47 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:43.867: INFO: 	Container kube-scheduler ready: true, restart count 0
Mar 16 19:43:43.867: INFO: ebs-csi-node-2dx52 started at 2023-03-16 19:31:13 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:43:43.867: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:43:43.867: INFO: cilium-operator-55dbd764f9-x6vkm started at 2023-03-16 19:31:13 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:43.867: INFO: 	Container cilium-operator ready: true, restart count 0
Mar 16 19:43:43.867: INFO: dns-controller-68cbff8fc-8fsm9 started at 2023-03-16 19:31:13 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:43.867: INFO: 	Container dns-controller ready: true, restart count 0
Mar 16 19:43:43.867: INFO: kops-controller-zttzd started at 2023-03-16 19:31:13 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:43.867: INFO: 	Container kops-controller ready: true, restart count 0
Mar 16 19:43:43.867: INFO: etcd-manager-events-i-006f135965e189394 started at 2023-03-16 19:28:47 +0000 UTC (11+1 container statuses recorded)
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 16 19:43:43.867: INFO: etcd-manager-main-i-006f135965e189394 started at 2023-03-16 19:28:47 +0000 UTC (11+1 container statuses recorded)
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 16 19:43:43.867: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 16 19:43:43.867: INFO: kube-controller-manager-i-006f135965e189394 started at 2023-03-16 19:28:47 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:43.867: INFO: 	Container kube-controller-manager ready: true, restart count 4
Mar 16 19:43:44.138: INFO: 
Latency metrics for node i-006f135965e189394
Mar 16 19:43:44.138: INFO: 
Logging node info for node i-02c2ffb2c76c516de
Mar 16 19:43:44.204: INFO: Node Info: &Node{ObjectMeta:{i-02c2ffb2c76c516de    a5a98d76-d961-492a-a0ae-b628a2af0a9b 29014 0 2023-03-16 19:32:43 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-02c2ffb2c76c516de kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-02c2ffb2c76c516de topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7164":"i-02c2ffb2c76c516de","ebs.csi.aws.com":"i-02c2ffb2c76c516de"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-16 19:32:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-16 19:32:43 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-16 19:32:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"2600:1f14:550:7f00:d8a4::/80\"":{}}}} } {kube-controller-manager Update v1 2023-03-16 19:32:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {kubelet Update v1 2023-03-16 19:32:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-03-16 19:43:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-16 19:43:43 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:2600:1f14:550:7f00:d8a4::/80,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-02c2ffb2c76c516de,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[2600:1f14:550:7f00:d8a4::/80],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027191296 0} {<nil>} 3932804Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922333696 0} {<nil>} 3830404Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-16 19:43:17 +0000 UTC,LastTransitionTime:2023-03-16 19:32:43 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-16 19:43:17 +0000 UTC,LastTransitionTime:2023-03-16 19:32:43 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-16 19:43:17 +0000 UTC,LastTransitionTime:2023-03-16 19:32:43 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-16 19:43:17 +0000 UTC,LastTransitionTime:2023-03-16 19:32:52 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:2600:1f14:550:7f00:3579:1da1:4725:2f37,},NodeAddress{Type:InternalDNS,Address:i-02c2ffb2c76c516de.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-02c2ffb2c76c516de.us-west-2.compute.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec28d06d73efdbb6fe07a02217c97a2a,SystemUUID:ec28d06d-73ef-dbb6-fe07-a02217c97a2a,BootID:73530b10-f85c-4a38-9eb8-a746f140455a,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.824+c84c8add70281c,KubeProxyVersion:v1.27.0-alpha.3.824+c84c8add70281c,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:67948896,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:25627801,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:5025027,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7164^a5430bac-c432-11ed-acb2-42475842dd50 kubernetes.io/csi/ebs.csi.aws.com^vol-024657759810a9d58],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7164^a5430bac-c432-11ed-acb2-42475842dd50,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-024657759810a9d58,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0e25d348d5a015a84,DevicePath:,},},Config:nil,},}
Mar 16 19:43:44.204: INFO: 
Logging kubelet events for node i-02c2ffb2c76c516de
Mar 16 19:43:44.275: INFO: 
Logging pods the kubelet thinks is on node i-02c2ffb2c76c516de
Mar 16 19:43:44.348: INFO: ss-1 started at 2023-03-16 19:43:40 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.348: INFO: 	Container webserver ready: false, restart count 0
Mar 16 19:43:44.348: INFO: cilium-rgmxw started at 2023-03-16 19:32:44 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:43:44.348: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 16 19:43:44.348: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 16 19:43:44.348: INFO: ebs-csi-controller-74c4b4f9d9-6rpx4 started at 2023-03-16 19:32:54 +0000 UTC (0+5 container statuses recorded)
Mar 16 19:43:44.348: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:43:44.348: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:43:44.348: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 16 19:43:44.348: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:43:44.348: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:43:44.348: INFO: ss2-0 started at 2023-03-16 19:43:36 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.348: INFO: 	Container webserver ready: true, restart count 0
Mar 16 19:43:44.348: INFO: inline-volume-tester-8cl88 started at 2023-03-16 19:42:12 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.348: INFO: 	Container csi-volume-tester ready: true, restart count 0
Mar 16 19:43:44.348: INFO: up-down-2-lvtqh started at 2023-03-16 19:43:38 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.348: INFO: 	Container up-down-2 ready: true, restart count 0
Mar 16 19:43:44.348: INFO: csi-hostpathplugin-0 started at 2023-03-16 19:42:10 +0000 UTC (0+7 container statuses recorded)
Mar 16 19:43:44.348: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:43:44.348: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:43:44.348: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 16 19:43:44.348: INFO: 	Container csi-snapshotter ready: true, restart count 0
Mar 16 19:43:44.348: INFO: 	Container hostpath ready: true, restart count 0
Mar 16 19:43:44.348: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:43:44.348: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:43:44.348: INFO: startup-4ccaea6b-c70b-4b6c-a78b-8374db6d2426 started at 2023-03-16 19:41:02 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.348: INFO: 	Container busybox ready: false, restart count 0
Mar 16 19:43:44.348: INFO: ebs-csi-node-dklhf started at 2023-03-16 19:32:44 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:43:44.348: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:43:44.348: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:43:44.348: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:43:44.348: INFO: oidc-discovery-validator started at 2023-03-16 19:43:37 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.348: INFO: 	Container oidc-discovery-validator ready: false, restart count 0
Mar 16 19:43:44.348: INFO: up-down-1-nwdtg started at 2023-03-16 19:43:35 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.348: INFO: 	Container up-down-1 ready: true, restart count 0
Mar 16 19:43:44.692: INFO: 
Latency metrics for node i-02c2ffb2c76c516de
Mar 16 19:43:44.692: INFO: 
Logging node info for node i-02ed4b9ad83b92b97
Mar 16 19:43:44.757: INFO: Node Info: &Node{ObjectMeta:{i-02ed4b9ad83b92b97    d9759ff2-9dc9-4ed2-9309-62f8c3e8e9f7 28982 0 2023-03-16 19:32:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-02ed4b9ad83b92b97 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-02ed4b9ad83b92b97 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02ed4b9ad83b92b97"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-03-16 19:32:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {kubelet Update v1 2023-03-16 19:32:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-16 19:32:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-16 19:32:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-16 19:32:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"2600:1f14:550:7f00:ae09::/80\"":{}}}} } {kubelet Update v1 2023-03-16 19:43:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-16 19:43:42 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:2600:1f14:550:7f00:ae09::/80,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-02ed4b9ad83b92b97,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[2600:1f14:550:7f00:ae09::/80],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027191296 0} {<nil>} 3932804Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922333696 0} {<nil>} 3830404Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-16 19:43:29 +0000 UTC,LastTransitionTime:2023-03-16 19:32:47 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-16 19:43:29 +0000 UTC,LastTransitionTime:2023-03-16 19:32:47 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-16 19:43:29 +0000 UTC,LastTransitionTime:2023-03-16 19:32:47 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-16 19:43:29 +0000 UTC,LastTransitionTime:2023-03-16 19:32:58 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:2600:1f14:550:7f00:1812:ef4a:9c80:6345,},NodeAddress{Type:InternalDNS,Address:i-02ed4b9ad83b92b97.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-02ed4b9ad83b92b97.us-west-2.compute.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2036eeaef2debf99dbb8a752e1b8c1,SystemUUID:ec2036ee-aef2-debf-99db-b8a752e1b8c1,BootID:6eee1f7e-fba6-494b-9d76-2fbfdff3eab5,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.824+c84c8add70281c,KubeProxyVersion:v1.27.0-alpha.3.824+c84c8add70281c,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:67948896,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:25627801,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0937aeabeed80244e],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0937aeabeed80244e,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0e9b20701e8e7b04c,DevicePath:,},},Config:nil,},}
Mar 16 19:43:44.758: INFO: 
Logging kubelet events for node i-02ed4b9ad83b92b97
Mar 16 19:43:44.826: INFO: 
Logging pods the kubelet thinks is on node i-02ed4b9ad83b92b97
Mar 16 19:43:44.897: INFO: coredns-745bddcfbb-hscd7 started at 2023-03-16 19:33:07 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container coredns ready: true, restart count 0
Mar 16 19:43:44.897: INFO: pod1-handle-http-request started at 2023-03-16 19:43:34 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container container-handle-8090-request ready: true, restart count 0
Mar 16 19:43:44.897: INFO: up-down-1-9nn8t started at 2023-03-16 19:43:35 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container up-down-1 ready: true, restart count 0
Mar 16 19:43:44.897: INFO: pod2-handle-http-request started at 2023-03-16 19:43:36 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container container-handle-8090-request ready: true, restart count 0
Mar 16 19:43:44.897: INFO: logs-generator started at 2023-03-16 19:43:43 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container logs-generator ready: true, restart count 0
Mar 16 19:43:44.897: INFO: verify-service-up-exec-pod-s88c8 started at 2023-03-16 19:43:44 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container agnhost-container ready: false, restart count 0
Mar 16 19:43:44.897: INFO: ss2-2 started at 2023-03-16 19:43:31 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container webserver ready: true, restart count 0
Mar 16 19:43:44.897: INFO: hostpath-symlink-prep-provisioning-1379 started at 2023-03-16 19:43:44 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container init-volume-provisioning-1379 ready: false, restart count 0
Mar 16 19:43:44.897: INFO: verify-service-up-host-exec-pod started at 2023-03-16 19:43:41 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 16 19:43:44.897: INFO: cilium-htdgf started at 2023-03-16 19:32:48 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 16 19:43:44.897: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 16 19:43:44.897: INFO: ss-0 started at 2023-03-16 19:43:28 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container webserver ready: true, restart count 0
Mar 16 19:43:44.897: INFO: startup-103bcbad-49d6-46fa-8703-462dea592977 started at 2023-03-16 19:43:18 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container busybox ready: false, restart count 0
Mar 16 19:43:44.897: INFO: hostexec-i-02ed4b9ad83b92b97-kn8z5 started at 2023-03-16 19:43:42 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 16 19:43:44.897: INFO: ebs-csi-node-74c5w started at 2023-03-16 19:32:48 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:43:44.897: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:43:44.897: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:43:44.897: INFO: pfpod started at 2023-03-16 19:43:32 +0000 UTC (0+2 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container portforwardtester ready: false, restart count 0
Mar 16 19:43:44.897: INFO: 	Container readiness ready: false, restart count 0
Mar 16 19:43:44.897: INFO: test-ss-0 started at 2023-03-16 19:43:37 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container webserver ready: true, restart count 0
Mar 16 19:43:44.897: INFO: ss-0 started at 2023-03-16 19:43:40 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:44.897: INFO: 	Container webserver ready: false, restart count 0
Mar 16 19:43:45.229: INFO: 
Latency metrics for node i-02ed4b9ad83b92b97
Mar 16 19:43:45.229: INFO: 
Logging node info for node i-07b8283b27a926ea4
Mar 16 19:43:45.294: INFO: Node Info: &Node{ObjectMeta:{i-07b8283b27a926ea4    2926fed8-06e5-47f1-80a5-351cf76c4d38 29077 0 2023-03-16 19:32:41 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-07b8283b27a926ea4 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-07b8283b27a926ea4 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-volume-expand-227":"i-07b8283b27a926ea4","ebs.csi.aws.com":"i-07b8283b27a926ea4"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-16 19:32:41 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-16 19:32:41 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-16 19:32:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kops-controller Update v1 2023-03-16 19:32:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"2600:1f14:550:7f00:182a::/80\"":{}}}} } {kube-controller-manager Update v1 2023-03-16 19:32:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {kubelet Update v1 2023-03-16 19:43:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:2600:1f14:550:7f00:182a::/80,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-07b8283b27a926ea4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[2600:1f14:550:7f00:182a::/80],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027187200 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922329600 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-16 19:43:44 +0000 UTC,LastTransitionTime:2023-03-16 19:32:41 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-16 19:43:44 +0000 UTC,LastTransitionTime:2023-03-16 19:32:41 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-16 19:43:44 +0000 UTC,LastTransitionTime:2023-03-16 19:32:41 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-16 19:43:44 +0000 UTC,LastTransitionTime:2023-03-16 19:32:51 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:2600:1f14:550:7f00:58e6:44e0:8dbc:9afe,},NodeAddress{Type:InternalDNS,Address:i-07b8283b27a926ea4.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-07b8283b27a926ea4.us-west-2.compute.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2014203de088435d46874c53715c1c,SystemUUID:ec201420-3de0-8843-5d46-874c53715c1c,BootID:9a378fc2-0572-4834-8107-b622fa5e0388,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.824+c84c8add70281c,KubeProxyVersion:v1.27.0-alpha.3.824+c84c8add70281c,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:67948896,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:25627801,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 16 19:43:45.295: INFO: 
Logging kubelet events for node i-07b8283b27a926ea4
Mar 16 19:43:45.363: INFO: 
Logging pods the kubelet thinks is on node i-07b8283b27a926ea4
Mar 16 19:43:45.507: INFO: coredns-745bddcfbb-bmht8 started at 2023-03-16 19:32:51 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:45.507: INFO: 	Container coredns ready: true, restart count 0
Mar 16 19:43:45.507: INFO: cilium-fs2kv started at 2023-03-16 19:32:42 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:43:45.507: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 16 19:43:45.507: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 16 19:43:45.507: INFO: pod-prestop-hook-c05a7dd4-c3ee-45e0-b630-75d56ea6c619 started at 2023-03-16 19:43:37 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:45.507: INFO: 	Container nginx ready: true, restart count 0
Mar 16 19:43:45.507: INFO: up-down-2-g5rks started at 2023-03-16 19:43:38 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:45.507: INFO: 	Container up-down-2 ready: true, restart count 0
Mar 16 19:43:45.507: INFO: ebs-csi-node-vqqvw started at 2023-03-16 19:32:42 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:43:45.507: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:43:45.507: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:43:45.507: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:43:45.507: INFO: coredns-autoscaler-8495f5575-8hwhz started at 2023-03-16 19:32:51 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:45.507: INFO: 	Container autoscaler ready: true, restart count 0
Mar 16 19:43:45.507: INFO: csi-hostpathplugin-0 started at 2023-03-16 19:43:20 +0000 UTC (0+7 container statuses recorded)
Mar 16 19:43:45.507: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:43:45.507: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:43:45.507: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 16 19:43:45.507: INFO: 	Container csi-snapshotter ready: true, restart count 0
Mar 16 19:43:45.507: INFO: 	Container hostpath ready: true, restart count 0
Mar 16 19:43:45.507: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:43:45.507: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:43:45.507: INFO: ebs-csi-controller-74c4b4f9d9-6l4cr started at 2023-03-16 19:32:51 +0000 UTC (0+5 container statuses recorded)
Mar 16 19:43:45.507: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:43:45.507: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:43:45.507: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 16 19:43:45.507: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:43:45.507: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:43:45.815: INFO: 
Latency metrics for node i-07b8283b27a926ea4
Mar 16 19:43:45.815: INFO: 
Logging node info for node i-0b5975e41089e600d
Mar 16 19:43:45.880: INFO: Node Info: &Node{ObjectMeta:{i-0b5975e41089e600d    13d7c5c9-ada0-4e0c-a9b4-6c3b0e64b4c3 28050 0 2023-03-16 19:32:53 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0b5975e41089e600d kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0b5975e41089e600d topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-expansion-2970":"csi-mock-csi-mock-volumes-expansion-2970","ebs.csi.aws.com":"i-0b5975e41089e600d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-16 19:32:53 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-16 19:32:53 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-16 19:32:53 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"2600:1f14:550:7f00:f985::/80\"":{}}}} } {kube-controller-manager Update v1 2023-03-16 19:32:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {kubelet Update v1 2023-03-16 19:32:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-16 19:43:19 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-16 19:43:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:2600:1f14:550:7f00:f985::/80,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0b5975e41089e600d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[2600:1f14:550:7f00:f985::/80],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027187200 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922329600 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-16 19:43:27 +0000 UTC,LastTransitionTime:2023-03-16 19:32:53 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-16 19:43:27 +0000 UTC,LastTransitionTime:2023-03-16 19:32:53 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-16 19:43:27 +0000 UTC,LastTransitionTime:2023-03-16 19:32:53 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-16 19:43:27 +0000 UTC,LastTransitionTime:2023-03-16 19:33:02 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:2600:1f14:550:7f00:c0aa:5f30:7328:8e3f,},NodeAddress{Type:InternalDNS,Address:i-0b5975e41089e600d.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0b5975e41089e600d.us-west-2.compute.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2173701442f95a91d69a4b0877f610,SystemUUID:ec217370-1442-f95a-91d6-9a4b0877f610,BootID:7b20997a-d9ac-4e40-b5c7-ba313b4b084b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.824+c84c8add70281c,KubeProxyVersion:v1.27.0-alpha.3.824+c84c8add70281c,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:67948896,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:25627801,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2003505,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-2970^4 kubernetes.io/csi/ebs.csi.aws.com^vol-01d3c1900b3344f99],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-01d3c1900b3344f99,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-2970^4,DevicePath:,},},Config:nil,},}
Mar 16 19:43:45.881: INFO: 
Logging kubelet events for node i-0b5975e41089e600d
Mar 16 19:43:45.956: INFO: 
Logging pods the kubelet thinks is on node i-0b5975e41089e600d
Mar 16 19:43:46.033: INFO: master started at 2023-03-16 19:43:25 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container cntr ready: true, restart count 0
Mar 16 19:43:46.033: INFO: hostexec-i-0b5975e41089e600d-f6zmz started at 2023-03-16 19:43:45 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container agnhost-container ready: false, restart count 0
Mar 16 19:43:46.033: INFO: pvc-volume-tester-cd8pq started at 2023-03-16 19:43:18 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container volume-tester ready: true, restart count 0
Mar 16 19:43:46.033: INFO: liveness-f01fa852-f2df-4fdb-9c5b-42a847a20912 started at 2023-03-16 19:42:11 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container agnhost-container ready: true, restart count 4
Mar 16 19:43:46.033: INFO: slave started at 2023-03-16 19:43:27 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container cntr ready: true, restart count 0
Mar 16 19:43:46.033: INFO: cilium-9vmm6 started at 2023-03-16 19:32:54 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 16 19:43:46.033: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 16 19:43:46.033: INFO: default started at 2023-03-16 19:43:32 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container cntr ready: true, restart count 0
Mar 16 19:43:46.033: INFO: ebs-csi-node-rkwb9 started at 2023-03-16 19:32:54 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:43:46.033: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:43:46.033: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:43:46.033: INFO: up-down-2-h7sn2 started at 2023-03-16 19:43:38 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container up-down-2 ready: true, restart count 0
Mar 16 19:43:46.033: INFO: csi-mockplugin-attacher-0 started at 2023-03-16 19:43:06 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:43:46.033: INFO: pod-handle-http-request started at 2023-03-16 19:41:26 +0000 UTC (0+2 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container container-handle-8090-request ready: true, restart count 0
Mar 16 19:43:46.033: INFO: 	Container container-handle-9090-request ready: true, restart count 0
Mar 16 19:43:46.033: INFO: test-grpc-87cfbfcc-15d7-4905-b834-3bdb16cd612d started at 2023-03-16 19:42:16 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container agnhost ready: true, restart count 0
Mar 16 19:43:46.033: INFO: test-webserver-b19c669d-3e70-4ef3-be05-eac5d97a6f20 started at 2023-03-16 19:42:45 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container test-webserver ready: false, restart count 0
Mar 16 19:43:46.033: INFO: private started at 2023-03-16 19:43:29 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container cntr ready: true, restart count 0
Mar 16 19:43:46.033: INFO: up-down-1-4g6jg started at 2023-03-16 19:43:35 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container up-down-1 ready: true, restart count 0
Mar 16 19:43:46.033: INFO: pause-pod-0 started at 2023-03-16 19:41:28 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 16 19:43:46.033: INFO: csi-mockplugin-0 started at 2023-03-16 19:43:06 +0000 UTC (0+4 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container busybox ready: true, restart count 0
Mar 16 19:43:46.033: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:43:46.033: INFO: 	Container driver-registrar ready: true, restart count 0
Mar 16 19:43:46.033: INFO: 	Container mock ready: true, restart count 0
Mar 16 19:43:46.033: INFO: ss2-1 started at 2023-03-16 19:43:33 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container webserver ready: false, restart count 0
Mar 16 19:43:46.033: INFO: pause-pod-0 started at 2023-03-16 19:43:38 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:43:46.033: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 16 19:43:46.376: INFO: 
Latency metrics for node i-0b5975e41089e600d
END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/16/23 19:43:46.376 (3.193s)
< Exit [DeferCleanup (Each)] [sig-network] EndpointSlice - dump namespaces | framework.go:209 @ 03/16/23 19:43:46.376 (3.193s)
> Enter [DeferCleanup (Each)] [sig-network] EndpointSlice - tear down framework | framework.go:206 @ 03/16/23 19:43:46.376
STEP: Destroying namespace "endpointslice-8251" for this suite. - test/e2e/framework/framework.go:351 @ 03/16/23 19:43:46.376
< Exit [DeferCleanup (Each)] [sig-network] EndpointSlice - tear down framework | framework.go:206 @ 03/16/23 19:43:46.445 (68ms)
> Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/16/23 19:43:46.445
< Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/16/23 19:43:46.445 (0s)

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

Kubernetes e2e suite [It] [sig-network] Services should serve endpoints on same port and different protocols 49s

go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-network\]\sServices\sshould\sserve\sendpoints\son\ssame\sport\sand\sdifferent\sprotocols$'

[FAILED] Timed out after 34.131s.
Expected
    <nil>: nil
not to be nil
In [It] at: test/e2e/network/service.go:3703 @ 03/16/23 19:40:11.343

				> Enter [BeforeEach] [sig-network] Services - set up framework | framework.go:191 @ 03/16/23 19:39:25.573
STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/16/23 19:39:25.573
Mar 16 19:39:25.573: INFO: >>> kubeConfig: /root/.kube/config
STEP: Building a namespace api object, basename services - test/e2e/framework/framework.go:250 @ 03/16/23 19:39:25.574
STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/16/23 19:39:25.762
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/16/23 19:39:25.884
< Exit [BeforeEach] [sig-network] Services - set up framework | framework.go:191 @ 03/16/23 19:39:26.009 (436ms)
> Enter [BeforeEach] [sig-network] Services - test/e2e/framework/metrics/init/init.go:33 @ 03/16/23 19:39:26.009
< Exit [BeforeEach] [sig-network] Services - test/e2e/framework/metrics/init/init.go:33 @ 03/16/23 19:39:26.009 (0s)
> Enter [BeforeEach] [sig-network] Services - test/e2e/network/service.go:764 @ 03/16/23 19:39:26.009
< Exit [BeforeEach] [sig-network] Services - test/e2e/network/service.go:764 @ 03/16/23 19:39:26.009 (0s)
> Enter [It] should serve endpoints on same port and different protocols - test/e2e/network/service.go:3622 @ 03/16/23 19:39:26.009
STEP: creating service multiprotocol-test in namespace services-2463 - test/e2e/network/service.go:3641 @ 03/16/23 19:39:26.009
STEP: creating pod pod1 in namespace services-2463 - test/e2e/network/service.go:3667 @ 03/16/23 19:39:26.082
STEP: Creating pod pod1 in namespace services-2463 - test/e2e/network/service.go:4094 @ 03/16/23 19:39:26.082
STEP: waiting up to 3m0s for service multiprotocol-test in namespace services-2463 to expose endpoints map[pod1:[{tcp-port 0 80 TCP } {udp-port 0 80 UDP }]] - test/e2e/network/service.go:4286 @ 03/16/23 19:39:28.346
Mar 16 19:39:28.596: INFO: successfully validated that service multiprotocol-test in namespace services-2463 exposes endpoints map[pod1:[{tcp-port 0 80 TCP } {udp-port 0 80 UDP }]]
STEP: Checking if the Service forwards traffic to the TCP and UDP port - test/e2e/network/service.go:3671 @ 03/16/23 19:39:28.596
Mar 16 19:39:28.596: INFO: Creating new exec pod
Mar 16 19:39:30.787: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=services-2463 exec execpoddwfvp -- /bin/sh -x -c echo hostName | nc -v -t -w 2 fd00:5e4f:ce::73a8 80'
Mar 16 19:39:31.552: INFO: stderr: "+ nc -v -t -w 2 fd00:5e4f:ce::73a8 80\n+ echo hostName\nConnection to fd00:5e4f:ce::73a8 80 port [tcp/http] succeeded!\n"
Mar 16 19:39:31.552: INFO: stdout: "HTTP/1.1 400 Bad Request\r\nContent-Type: text/plain; charset=utf-8\r\nConnection: close\r\n\r\n400 Bad Request"
Mar 16 19:39:31.552: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=services-2463 exec execpoddwfvp -- /bin/sh -x -c echo hostName | nc -v -u -w 2 fd00:5e4f:ce::73a8 80'
Mar 16 19:39:36.309: INFO: stderr: "+ nc -v -u -w 2 fd00:5e4f:ce::73a8 80\n+ echo hostName\nConnection to fd00:5e4f:ce::73a8 80 port [udp/*] succeeded!\n"
Mar 16 19:39:36.309: INFO: stdout: "pod1"
STEP: Checking if the Service forwards traffic to TCP only - test/e2e/network/service.go:3682 @ 03/16/23 19:39:36.309
Mar 16 19:39:36.439: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=services-2463 exec execpoddwfvp -- /bin/sh -x -c echo hostName | nc -v -t -w 2 fd00:5e4f:ce::73a8 80'
Mar 16 19:39:37.212: INFO: stderr: "+ echo hostName\n+ nc -v -t -w 2 fd00:5e4f:ce::73a8 80\nConnection to fd00:5e4f:ce::73a8 80 port [tcp/http] succeeded!\n"
Mar 16 19:39:37.212: INFO: stdout: "HTTP/1.1 400 Bad Request\r\nContent-Type: text/plain; charset=utf-8\r\nConnection: close\r\n\r\n400 Bad Request"
Mar 16 19:39:37.212: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=services-2463 exec execpoddwfvp -- /bin/sh -x -c echo hostName | nc -v -u -w 2 fd00:5e4f:ce::73a8 80'
Mar 16 19:39:41.991: INFO: stderr: "+ echo hostName\n+ nc -v -u -w 2 fd00:5e4f:ce::73a8 80\nConnection to fd00:5e4f:ce::73a8 80 port [udp/*] succeeded!\n"
Mar 16 19:39:41.991: INFO: stdout: "pod1"
Mar 16 19:39:46.992: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=services-2463 exec execpoddwfvp -- /bin/sh -x -c echo hostName | nc -v -u -w 2 fd00:5e4f:ce::73a8 80'
Mar 16 19:39:51.762: INFO: stderr: "+ nc -v -u -w 2 fd00:5e4f:ce::73a8 80\n+ echo hostName\nConnection to fd00:5e4f:ce::73a8 80 port [udp/*] succeeded!\n"
Mar 16 19:39:51.762: INFO: stdout: "pod1"
Mar 16 19:39:56.763: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=services-2463 exec execpoddwfvp -- /bin/sh -x -c echo hostName | nc -v -u -w 2 fd00:5e4f:ce::73a8 80'
Mar 16 19:40:01.581: INFO: stderr: "+ echo hostName\n+ nc -v -u -w 2 fd00:5e4f:ce::73a8 80\nConnection to fd00:5e4f:ce::73a8 80 port [udp/*] succeeded!\n"
Mar 16 19:40:01.581: INFO: stdout: "pod1"
Mar 16 19:40:06.581: INFO: Running '/home/prow/go/src/k8s.io/kops/_rundir/ede149a1-c42f-11ed-93d6-92b4ce3fddda/kubectl --server=https://api-e2e-e2e-kops-aws-cni--jivt6e-92590bb8d15091e3.elb.us-west-2.amazonaws.com --kubeconfig=/root/.kube/config --namespace=services-2463 exec execpoddwfvp -- /bin/sh -x -c echo hostName | nc -v -u -w 2 fd00:5e4f:ce::73a8 80'
Mar 16 19:40:11.343: INFO: stderr: "+ nc -v -u -w 2 fd00:5e4f:ce::73a8 80\n+ echo hostName\nConnection to fd00:5e4f:ce::73a8 80 port [udp/*] succeeded!\n"
Mar 16 19:40:11.343: INFO: stdout: "pod1"
[FAILED] Timed out after 34.131s.
Expected
    <nil>: nil
not to be nil
In [It] at: test/e2e/network/service.go:3703 @ 03/16/23 19:40:11.343
< Exit [It] should serve endpoints on same port and different protocols - test/e2e/network/service.go:3622 @ 03/16/23 19:40:11.343 (45.334s)
> Enter [AfterEach] [sig-network] Services - test/e2e/framework/node/init/init.go:33 @ 03/16/23 19:40:11.343
Mar 16 19:40:11.343: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
< Exit [AfterEach] [sig-network] Services - test/e2e/framework/node/init/init.go:33 @ 03/16/23 19:40:11.405 (63ms)
> Enter [DeferCleanup (Each)] [sig-network] Services - test/e2e/framework/metrics/init/init.go:35 @ 03/16/23 19:40:11.405
< Exit [DeferCleanup (Each)] [sig-network] Services - test/e2e/framework/metrics/init/init.go:35 @ 03/16/23 19:40:11.406 (0s)
> Enter [DeferCleanup (Each)] [sig-network] Services - dump namespaces | framework.go:209 @ 03/16/23 19:40:11.406
STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/16/23 19:40:11.406
STEP: Collecting events from namespace "services-2463". - test/e2e/framework/debug/dump.go:42 @ 03/16/23 19:40:11.406
STEP: Found 8 events. - test/e2e/framework/debug/dump.go:46 @ 03/16/23 19:40:11.468
Mar 16 19:40:11.468: INFO: At 2023-03-16 19:39:26 +0000 UTC - event for pod1: {default-scheduler } Scheduled: Successfully assigned services-2463/pod1 to i-02c2ffb2c76c516de
Mar 16 19:40:11.468: INFO: At 2023-03-16 19:39:26 +0000 UTC - event for pod1: {kubelet i-02c2ffb2c76c516de} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Mar 16 19:40:11.468: INFO: At 2023-03-16 19:39:26 +0000 UTC - event for pod1: {kubelet i-02c2ffb2c76c516de} Created: Created container agnhost-container
Mar 16 19:40:11.468: INFO: At 2023-03-16 19:39:26 +0000 UTC - event for pod1: {kubelet i-02c2ffb2c76c516de} Started: Started container agnhost-container
Mar 16 19:40:11.468: INFO: At 2023-03-16 19:39:28 +0000 UTC - event for execpoddwfvp: {default-scheduler } Scheduled: Successfully assigned services-2463/execpoddwfvp to i-0b5975e41089e600d
Mar 16 19:40:11.468: INFO: At 2023-03-16 19:39:29 +0000 UTC - event for execpoddwfvp: {kubelet i-0b5975e41089e600d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine
Mar 16 19:40:11.468: INFO: At 2023-03-16 19:39:29 +0000 UTC - event for execpoddwfvp: {kubelet i-0b5975e41089e600d} Created: Created container agnhost-container
Mar 16 19:40:11.468: INFO: At 2023-03-16 19:39:29 +0000 UTC - event for execpoddwfvp: {kubelet i-0b5975e41089e600d} Started: Started container agnhost-container
Mar 16 19:40:11.538: INFO: POD           NODE                 PHASE    GRACE  CONDITIONS
Mar 16 19:40:11.538: INFO: execpoddwfvp  i-0b5975e41089e600d  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:39:28 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:39:30 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:39:30 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:39:28 +0000 UTC  }]
Mar 16 19:40:11.538: INFO: pod1          i-02c2ffb2c76c516de  Running         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:39:26 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:39:27 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:39:27 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-16 19:39:26 +0000 UTC  }]
Mar 16 19:40:11.539: INFO: 
Mar 16 19:40:11.744: INFO: 
Logging node info for node i-006f135965e189394
Mar 16 19:40:11.806: INFO: Node Info: &Node{ObjectMeta:{i-006f135965e189394    f587fccb-3521-41bc-a111-b85c50902da0 10506 0 2023-03-16 19:30:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:c6g.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:arm64 kubernetes.io/hostname:i-006f135965e189394 kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c6g.large topology.ebs.csi.aws.com/zone:us-west-2a topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-006f135965e189394"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] []},Spec:NodeSpec{PodCIDR:2600:1f14:550:7f01:ff27::/80,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-006f135965e189394,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[2600:1f14:550:7f01:ff27::/80],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4002074624 0} {<nil>} 3908276Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3897217024 0} {<nil>} 3805876Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-16 19:38:02 +0000 UTC,LastTransitionTime:2023-03-16 19:30:31 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-16 19:38:02 +0000 UTC,LastTransitionTime:2023-03-16 19:30:31 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-16 19:38:02 +0000 UTC,LastTransitionTime:2023-03-16 19:30:31 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-16 19:38:02 +0000 UTC,LastTransitionTime:2023-03-16 19:31:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:2600:1f14:550:7f01:2dc8:a998:6a95:8a0,},NodeAddress{Type:InternalIP,Address:172.20.44.142,},NodeAddress{Type:InternalDNS,Address:i-006f135965e189394.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-006f135965e189394.us-west-2.compute.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec23b983e7b3e74caa3587361d845f96,SystemUUID:ec23b983-e7b3-e74c-aa35-87361d845f96,BootID:f8440a9c-092a-400d-a2ee-e2e49acbe0b8,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.824+c84c8add70281c,KubeProxyVersion:v1.27.0-alpha.3.824+c84c8add70281c,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:157797353,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:151677080,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:134531559,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:126979107,},ContainerImage{Names:[registry.k8s.io/kube-apiserver-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:115844451,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:108504573,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:81436097,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:81120118,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:81114891,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:80539316,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:67948896,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:61500433,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:60826655,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:59553904,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:57386480,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:41884696,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:40375541,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[quay.io/cilium/operator@sha256:a6d24a006a6b92967ac90786b49bc1ac26e5477cf028cd1186efcfc2466484db quay.io/cilium/operator:v1.12.5],SizeBytes:24567287,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:18345226,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:4740268,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 16 19:40:11.806: INFO: 
Logging kubelet events for node i-006f135965e189394
Mar 16 19:40:11.871: INFO: 
Logging pods the kubelet thinks is on node i-006f135965e189394
Mar 16 19:40:12.000: INFO: kube-controller-manager-i-006f135965e189394 started at 2023-03-16 19:28:47 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.000: INFO: 	Container kube-controller-manager ready: true, restart count 4
Mar 16 19:40:12.000: INFO: cilium-operator-55dbd764f9-x6vkm started at 2023-03-16 19:31:13 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.000: INFO: 	Container cilium-operator ready: true, restart count 0
Mar 16 19:40:12.000: INFO: dns-controller-68cbff8fc-8fsm9 started at 2023-03-16 19:31:13 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.000: INFO: 	Container dns-controller ready: true, restart count 0
Mar 16 19:40:12.000: INFO: kops-controller-zttzd started at 2023-03-16 19:31:13 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.000: INFO: 	Container kops-controller ready: true, restart count 0
Mar 16 19:40:12.000: INFO: etcd-manager-events-i-006f135965e189394 started at 2023-03-16 19:28:47 +0000 UTC (11+1 container statuses recorded)
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 16 19:40:12.000: INFO: etcd-manager-main-i-006f135965e189394 started at 2023-03-16 19:28:47 +0000 UTC (11+1 container statuses recorded)
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-2-24 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-3-10 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-3-17 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-4-13 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-4-3 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-5-0 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-5-1 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-5-3 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-5-4 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-5-6 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Init container init-etcd-3-5-7 ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Container etcd-manager ready: true, restart count 0
Mar 16 19:40:12.000: INFO: ebs-csi-node-2dx52 started at 2023-03-16 19:31:13 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:40:12.000: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:40:12.000: INFO: cilium-d786k started at 2023-03-16 19:31:13 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:40:12.000: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 16 19:40:12.000: INFO: aws-cloud-controller-manager-2zl7t started at 2023-03-16 19:31:13 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.000: INFO: 	Container aws-cloud-controller-manager ready: true, restart count 0
Mar 16 19:40:12.000: INFO: kube-apiserver-i-006f135965e189394 started at 2023-03-16 19:28:47 +0000 UTC (0+2 container statuses recorded)
Mar 16 19:40:12.000: INFO: 	Container healthcheck ready: true, restart count 0
Mar 16 19:40:12.000: INFO: 	Container kube-apiserver ready: true, restart count 3
Mar 16 19:40:12.000: INFO: kube-scheduler-i-006f135965e189394 started at 2023-03-16 19:28:47 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.000: INFO: 	Container kube-scheduler ready: true, restart count 0
Mar 16 19:40:12.270: INFO: 
Latency metrics for node i-006f135965e189394
Mar 16 19:40:12.270: INFO: 
Logging node info for node i-02c2ffb2c76c516de
Mar 16 19:40:12.335: INFO: Node Info: &Node{ObjectMeta:{i-02c2ffb2c76c516de    a5a98d76-d961-492a-a0ae-b628a2af0a9b 16688 0 2023-03-16 19:32:43 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-02c2ffb2c76c516de kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-02c2ffb2c76c516de topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-3049":"i-02c2ffb2c76c516de","csi-hostpath-ephemeral-8182":"i-02c2ffb2c76c516de","csi-hostpath-ephemeral-8889":"i-02c2ffb2c76c516de","csi-hostpath-provisioning-5728":"i-02c2ffb2c76c516de","ebs.csi.aws.com":"i-02c2ffb2c76c516de"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-16 19:32:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-16 19:32:43 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-16 19:32:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"2600:1f14:550:7f00:d8a4::/80\"":{}}}} } {kube-controller-manager Update v1 2023-03-16 19:32:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {kubelet Update v1 2023-03-16 19:32:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kubelet Update v1 2023-03-16 19:40:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-16 19:40:05 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:2600:1f14:550:7f00:d8a4::/80,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-02c2ffb2c76c516de,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[2600:1f14:550:7f00:d8a4::/80],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027191296 0} {<nil>} 3932804Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922333696 0} {<nil>} 3830404Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-16 19:40:02 +0000 UTC,LastTransitionTime:2023-03-16 19:32:43 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-16 19:40:02 +0000 UTC,LastTransitionTime:2023-03-16 19:32:43 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-16 19:40:02 +0000 UTC,LastTransitionTime:2023-03-16 19:32:43 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-16 19:40:02 +0000 UTC,LastTransitionTime:2023-03-16 19:32:52 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:2600:1f14:550:7f00:3579:1da1:4725:2f37,},NodeAddress{Type:InternalDNS,Address:i-02c2ffb2c76c516de.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-02c2ffb2c76c516de.us-west-2.compute.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec28d06d73efdbb6fe07a02217c97a2a,SystemUUID:ec28d06d-73ef-dbb6-fe07-a02217c97a2a,BootID:73530b10-f85c-4a38-9eb8-a746f140455a,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.824+c84c8add70281c,KubeProxyVersion:v1.27.0-alpha.3.824+c84c8add70281c,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:67948896,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:25627801,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-3049^48270454-c432-11ed-a196-c271cf08b81b kubernetes.io/csi/csi-hostpath-ephemeral-3049^4e1e0253-c432-11ed-a196-c271cf08b81b kubernetes.io/csi/csi-hostpath-ephemeral-8889^54ae81de-c432-11ed-a1c9-8222d7472f8f],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-3049^48270454-c432-11ed-a196-c271cf08b81b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-3049^4e1e0253-c432-11ed-a196-c271cf08b81b,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-8889^54ae81de-c432-11ed-a1c9-8222d7472f8f,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-5728^58f765ca-c432-11ed-bd10-1ad2a54a4d42,DevicePath:,},},Config:nil,},}
Mar 16 19:40:12.336: INFO: 
Logging kubelet events for node i-02c2ffb2c76c516de
Mar 16 19:40:12.413: INFO: 
Logging pods the kubelet thinks is on node i-02c2ffb2c76c516de
Mar 16 19:40:12.546: INFO: cilium-rgmxw started at 2023-03-16 19:32:44 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 16 19:40:12.546: INFO: ebs-csi-controller-74c4b4f9d9-6rpx4 started at 2023-03-16 19:32:54 +0000 UTC (0+5 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:40:12.546: INFO: inline-volume-tester-t2vpq started at 2023-03-16 19:39:36 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container csi-volume-tester ready: true, restart count 0
Mar 16 19:40:12.546: INFO: pod-subpath-test-dynamicpv-8p5r started at 2023-03-16 19:40:04 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Init container init-volume-dynamicpv-8p5r ready: false, restart count 0
Mar 16 19:40:12.546: INFO: 	Container test-container-subpath-dynamicpv-8p5r ready: false, restart count 0
Mar 16 19:40:12.546: INFO: inline-volume-tester-6f4j8 started at 2023-03-16 19:39:57 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container csi-volume-tester ready: true, restart count 0
Mar 16 19:40:12.546: INFO: csi-mockplugin-attacher-0 started at 2023-03-16 19:40:11 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container csi-attacher ready: false, restart count 0
Mar 16 19:40:12.546: INFO: inline-volume-tester-r7x7r started at 2023-03-16 19:39:45 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container csi-volume-tester ready: true, restart count 0
Mar 16 19:40:12.546: INFO: affinity-clusterip-transition-7vjts started at 2023-03-16 19:40:07 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container affinity-clusterip-transition ready: true, restart count 0
Mar 16 19:40:12.546: INFO: ebs-csi-node-dklhf started at 2023-03-16 19:32:44 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:40:12.546: INFO: nodeport-test-zwn5m started at 2023-03-16 19:39:55 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container nodeport-test ready: true, restart count 0
Mar 16 19:40:12.546: INFO: csi-mockplugin-0 started at 2023-03-16 19:40:11 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container csi-provisioner ready: false, restart count 0
Mar 16 19:40:12.546: INFO: 	Container driver-registrar ready: false, restart count 0
Mar 16 19:40:12.546: INFO: 	Container mock ready: false, restart count 0
Mar 16 19:40:12.546: INFO: webserver-pod started at 2023-03-16 19:36:17 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container agnhost ready: false, restart count 0
Mar 16 19:40:12.546: INFO: csi-hostpathplugin-0 started at 2023-03-16 19:40:02 +0000 UTC (0+7 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container csi-snapshotter ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container hostpath ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:40:12.546: INFO: inline-volume-tester2-nc8vx started at 2023-03-16 19:39:46 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container csi-volume-tester ready: true, restart count 0
Mar 16 19:40:12.546: INFO: pause-pod-1 started at 2023-03-16 19:39:20 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 16 19:40:12.546: INFO: pod1 started at 2023-03-16 19:39:26 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 16 19:40:12.546: INFO: csi-hostpathplugin-0 started at 2023-03-16 19:39:55 +0000 UTC (0+7 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container csi-snapshotter ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container hostpath ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:40:12.546: INFO: csi-hostpathplugin-0 started at 2023-03-16 19:39:33 +0000 UTC (0+7 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container csi-snapshotter ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container hostpath ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:40:12.546: INFO: csi-hostpathplugin-0 started at 2023-03-16 19:39:45 +0000 UTC (0+7 container statuses recorded)
Mar 16 19:40:12.546: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container csi-snapshotter ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container hostpath ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:40:12.546: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:40:12.936: INFO: 
Latency metrics for node i-02c2ffb2c76c516de
Mar 16 19:40:12.936: INFO: 
Logging node info for node i-02ed4b9ad83b92b97
Mar 16 19:40:13.002: INFO: Node Info: &Node{ObjectMeta:{i-02ed4b9ad83b92b97    d9759ff2-9dc9-4ed2-9309-62f8c3e8e9f7 16719 0 2023-03-16 19:32:47 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-02ed4b9ad83b92b97 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-02ed4b9ad83b92b97 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02ed4b9ad83b92b97"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kube-controller-manager Update v1 2023-03-16 19:32:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {kubelet Update v1 2023-03-16 19:32:47 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {aws-cloud-controller-manager Update v1 2023-03-16 19:32:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-16 19:32:48 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-16 19:32:48 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"2600:1f14:550:7f00:ae09::/80\"":{}}}} } {kube-controller-manager Update v1 2023-03-16 19:40:06 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-16 19:40:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:2600:1f14:550:7f00:ae09::/80,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-02ed4b9ad83b92b97,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[2600:1f14:550:7f00:ae09::/80],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027191296 0} {<nil>} 3932804Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922333696 0} {<nil>} 3830404Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-16 19:40:06 +0000 UTC,LastTransitionTime:2023-03-16 19:32:47 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-16 19:40:06 +0000 UTC,LastTransitionTime:2023-03-16 19:32:47 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-16 19:40:06 +0000 UTC,LastTransitionTime:2023-03-16 19:32:47 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-16 19:40:06 +0000 UTC,LastTransitionTime:2023-03-16 19:32:58 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:2600:1f14:550:7f00:1812:ef4a:9c80:6345,},NodeAddress{Type:InternalDNS,Address:i-02ed4b9ad83b92b97.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-02ed4b9ad83b92b97.us-west-2.compute.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2036eeaef2debf99dbb8a752e1b8c1,SystemUUID:ec2036ee-aef2-debf-99db-b8a752e1b8c1,BootID:6eee1f7e-fba6-494b-9d76-2fbfdff3eab5,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.824+c84c8add70281c,KubeProxyVersion:v1.27.0-alpha.3.824+c84c8add70281c,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:67948896,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:25627801,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-07bb566bb6f83698a kubernetes.io/csi/ebs.csi.aws.com^vol-0825025a1768d4498],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-07bb566bb6f83698a,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0825025a1768d4498,DevicePath:,},},Config:nil,},}
Mar 16 19:40:13.002: INFO: 
Logging kubelet events for node i-02ed4b9ad83b92b97
Mar 16 19:40:13.084: INFO: 
Logging pods the kubelet thinks is on node i-02ed4b9ad83b92b97
Mar 16 19:40:13.155: INFO: nodeport-test-xf99z started at 2023-03-16 19:39:55 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.155: INFO: 	Container nodeport-test ready: true, restart count 0
Mar 16 19:40:13.155: INFO: inline-volume-tester-4gjkc started at 2023-03-16 19:39:58 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.155: INFO: 	Container csi-volume-tester ready: true, restart count 0
Mar 16 19:40:13.155: INFO: successful-jobs-history-limit-27983260-qsmlb started at 2023-03-16 19:40:00 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.155: INFO: 	Container c ready: false, restart count 0
Mar 16 19:40:13.155: INFO: test-webserver-9759ef91-7cad-4ad4-b31d-4b7ef8e410ec started at 2023-03-16 19:36:21 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.155: INFO: 	Container test-webserver ready: true, restart count 0
Mar 16 19:40:13.155: INFO: cilium-htdgf started at 2023-03-16 19:32:48 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:40:13.155: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 16 19:40:13.155: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 16 19:40:13.155: INFO: execpodwdxkq started at 2023-03-16 19:39:58 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.155: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 16 19:40:13.155: INFO: liveness-801324f7-9fce-414e-a584-95696114b87a started at 2023-03-16 19:38:53 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.155: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 16 19:40:13.155: INFO: pod-5fca509d-bd03-45ee-ba49-f03b4d36837f started at 2023-03-16 19:40:12 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.155: INFO: 	Container test-container ready: false, restart count 0
Mar 16 19:40:13.155: INFO: ebs-csi-node-74c5w started at 2023-03-16 19:32:48 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:40:13.155: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:40:13.155: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:40:13.155: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:40:13.155: INFO: hostexec-i-02ed4b9ad83b92b97-tmb87 started at 2023-03-16 19:40:06 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.155: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 16 19:40:13.155: INFO: affinity-clusterip-transition-d2wf5 started at 2023-03-16 19:40:07 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.155: INFO: 	Container affinity-clusterip-transition ready: true, restart count 0
Mar 16 19:40:13.155: INFO: coredns-745bddcfbb-hscd7 started at 2023-03-16 19:33:07 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.155: INFO: 	Container coredns ready: true, restart count 0
Mar 16 19:40:13.155: INFO: hostexec-i-02ed4b9ad83b92b97-pb7sx started at 2023-03-16 19:39:58 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.155: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 16 19:40:13.155: INFO: inline-volume-tester-vng27 started at 2023-03-16 19:39:40 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.155: INFO: 	Container csi-volume-tester ready: true, restart count 0
Mar 16 19:40:13.460: INFO: 
Latency metrics for node i-02ed4b9ad83b92b97
Mar 16 19:40:13.460: INFO: 
Logging node info for node i-07b8283b27a926ea4
Mar 16 19:40:13.524: INFO: Node Info: &Node{ObjectMeta:{i-07b8283b27a926ea4    2926fed8-06e5-47f1-80a5-351cf76c4d38 17094 0 2023-03-16 19:32:41 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a kubernetes.io/arch:arm64 kubernetes.io/hostname:i-07b8283b27a926ea4 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-07b8283b27a926ea4 topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-3370":"i-07b8283b27a926ea4","csi-hostpath-provisioning-2677":"i-07b8283b27a926ea4","csi-mock-csi-mock-volumes-workload-4646":"i-07b8283b27a926ea4","ebs.csi.aws.com":"i-07b8283b27a926ea4"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-16 19:32:41 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-16 19:32:41 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-16 19:32:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kops-controller Update v1 2023-03-16 19:32:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"2600:1f14:550:7f00:182a::/80\"":{}}}} } {kube-controller-manager Update v1 2023-03-16 19:32:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {kubelet Update v1 2023-03-16 19:40:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:2600:1f14:550:7f00:182a::/80,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-07b8283b27a926ea4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[2600:1f14:550:7f00:182a::/80],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027187200 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922329600 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-16 19:40:11 +0000 UTC,LastTransitionTime:2023-03-16 19:32:41 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-16 19:40:11 +0000 UTC,LastTransitionTime:2023-03-16 19:32:41 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-16 19:40:11 +0000 UTC,LastTransitionTime:2023-03-16 19:32:41 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-16 19:40:11 +0000 UTC,LastTransitionTime:2023-03-16 19:32:51 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:2600:1f14:550:7f00:58e6:44e0:8dbc:9afe,},NodeAddress{Type:InternalDNS,Address:i-07b8283b27a926ea4.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-07b8283b27a926ea4.us-west-2.compute.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2014203de088435d46874c53715c1c,SystemUUID:ec201420-3de0-8843-5d46-874c53715c1c,BootID:9a378fc2-0572-4834-8107-b622fa5e0388,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.824+c84c8add70281c,KubeProxyVersion:v1.27.0-alpha.3.824+c84c8add70281c,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:67948896,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:25627801,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:21735644,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:20848413,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:20565480,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:19007623,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:13766234,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
Mar 16 19:40:13.524: INFO: 
Logging kubelet events for node i-07b8283b27a926ea4
Mar 16 19:40:13.595: INFO: 
Logging pods the kubelet thinks is on node i-07b8283b27a926ea4
Mar 16 19:40:13.669: INFO: coredns-autoscaler-8495f5575-8hwhz started at 2023-03-16 19:32:51 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.669: INFO: 	Container autoscaler ready: true, restart count 0
Mar 16 19:40:13.669: INFO: csi-hostpathplugin-0 started at 2023-03-16 19:40:12 +0000 UTC (0+7 container statuses recorded)
Mar 16 19:40:13.669: INFO: 	Container csi-attacher ready: false, restart count 0
Mar 16 19:40:13.669: INFO: 	Container csi-provisioner ready: false, restart count 0
Mar 16 19:40:13.669: INFO: 	Container csi-resizer ready: false, restart count 0
Mar 16 19:40:13.669: INFO: 	Container csi-snapshotter ready: false, restart count 0
Mar 16 19:40:13.669: INFO: 	Container hostpath ready: false, restart count 0
Mar 16 19:40:13.669: INFO: 	Container liveness-probe ready: false, restart count 0
Mar 16 19:40:13.669: INFO: 	Container node-driver-registrar ready: false, restart count 0
Mar 16 19:40:13.669: INFO: pod-subpath-test-preprovisionedpv-2dqm started at 2023-03-16 19:40:13 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:40:13.669: INFO: 	Init container init-volume-preprovisionedpv-2dqm ready: false, restart count 0
Mar 16 19:40:13.669: INFO: 	Container test-container-subpath-preprovisionedpv-2dqm ready: false, restart count 0
Mar 16 19:40:13.669: INFO: csi-hostpathplugin-0 started at 2023-03-16 19:37:56 +0000 UTC (0+7 container statuses recorded)
Mar 16 19:40:13.669: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:40:13.669: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:40:13.669: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 16 19:40:13.669: INFO: 	Container csi-snapshotter ready: true, restart count 0
Mar 16 19:40:13.669: INFO: 	Container hostpath ready: true, restart count 0
Mar 16 19:40:13.669: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:40:13.669: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:40:13.669: INFO: ebs-csi-controller-74c4b4f9d9-6l4cr started at 2023-03-16 19:32:51 +0000 UTC (0+5 container statuses recorded)
Mar 16 19:40:13.669: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:40:13.669: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:40:13.669: INFO: 	Container csi-resizer ready: true, restart count 0
Mar 16 19:40:13.669: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:40:13.669: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:40:13.669: INFO: pvc-volume-tester-6546c started at 2023-03-16 19:39:31 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.669: INFO: 	Container volume-tester ready: true, restart count 0
Mar 16 19:40:13.669: INFO: coredns-745bddcfbb-bmht8 started at 2023-03-16 19:32:51 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.669: INFO: 	Container coredns ready: true, restart count 0
Mar 16 19:40:13.669: INFO: cilium-fs2kv started at 2023-03-16 19:32:42 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:40:13.669: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 16 19:40:13.669: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 16 19:40:13.669: INFO: csi-mockplugin-0 started at 2023-03-16 19:39:25 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:40:13.669: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:40:13.669: INFO: 	Container driver-registrar ready: true, restart count 0
Mar 16 19:40:13.669: INFO: 	Container mock ready: true, restart count 0
Mar 16 19:40:13.669: INFO: csi-mockplugin-attacher-0 started at 2023-03-16 19:39:26 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.669: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:40:13.669: INFO: ebs-csi-node-vqqvw started at 2023-03-16 19:32:42 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:40:13.669: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:40:13.669: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:40:13.669: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:40:13.669: INFO: hostexec-i-07b8283b27a926ea4-t6p8m started at 2023-03-16 19:40:00 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:13.669: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 16 19:40:13.997: INFO: 
Latency metrics for node i-07b8283b27a926ea4
Mar 16 19:40:13.997: INFO: 
Logging node info for node i-0b5975e41089e600d
Mar 16 19:40:14.062: INFO: Node Info: &Node{ObjectMeta:{i-0b5975e41089e600d    13d7c5c9-ada0-4e0c-a9b4-6c3b0e64b4c3 17060 0 2023-03-16 19:32:53 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:arm64 beta.kubernetes.io/instance-type:t4g.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:arm64 kubernetes.io/hostname:i-0b5975e41089e600d kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t4g.medium topology.ebs.csi.aws.com/zone:us-west-2a topology.hostpath.csi/node:i-0b5975e41089e600d topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-capacity-3059":"csi-mock-csi-mock-volumes-capacity-3059","csi-mock-csi-mock-volumes-capacity-5597":"i-0b5975e41089e600d","ebs.csi.aws.com":"i-0b5975e41089e600d"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-16 19:32:53 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-16 19:32:53 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-16 19:32:53 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"2600:1f14:550:7f00:f985::/80\"":{}}}} } {kube-controller-manager Update v1 2023-03-16 19:32:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {kubelet Update v1 2023-03-16 19:32:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-16 19:40:06 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-16 19:40:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:2600:1f14:550:7f00:f985::/80,DoNotUseExternalID:,ProviderID:aws:///us-west-2a/i-0b5975e41089e600d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[2600:1f14:550:7f00:f985::/80],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49756430336 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{4027187200 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44780787229 0} {<nil>} 44780787229 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-32Mi: {{0 0} {<nil>} 0 DecimalSI},hugepages-64Ki: {{0 0} {<nil>} 0 DecimalSI},memory: {{3922329600 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-16 19:40:12 +0000 UTC,LastTransitionTime:2023-03-16 19:32:53 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-16 19:40:12 +0000 UTC,LastTransitionTime:2023-03-16 19:32:53 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-16 19:40:12 +0000 UTC,LastTransitionTime:2023-03-16 19:32:53 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-16 19:40:12 +0000 UTC,LastTransitionTime:2023-03-16 19:33:02 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:2600:1f14:550:7f00:c0aa:5f30:7328:8e3f,},NodeAddress{Type:InternalDNS,Address:i-0b5975e41089e600d.us-west-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0b5975e41089e600d.us-west-2.compute.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2173701442f95a91d69a4b0877f610,SystemUUID:ec217370-1442-f95a-91d6-9a4b0877f610,BootID:7b20997a-d9ac-4e40-b5c7-ba313b4b084b,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 22.04.2 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.27.0-alpha.3.824+c84c8add70281c,KubeProxyVersion:v1.27.0-alpha.3.824+c84c8add70281c,OperatingSystem:linux,Architecture:arm64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:06ce2b0a0a472e73334a7504ee5c5d8b2e2d7b72ef728ad94e564740dd505be5 quay.io/cilium/cilium:v1.12.5],SizeBytes:157636062,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:116764683,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:80665728,},ContainerImage{Names:[registry.k8s.io/kube-proxy-arm64:v1.27.0-alpha.3.824_c84c8add70281c],SizeBytes:67948896,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:49808364,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41528818,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40343601,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:28932904,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:25627801,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:23857462,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:22678632,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:22434180,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:22394501,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18267611,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18159085,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:8502345,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8223819,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:7685305,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:6883347,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6860089,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:773367,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:772933,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:268051,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:253553,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-capacity-5597^59cc8a6b-c432-11ed-8866-0a147d529a4c],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-capacity-5597^59cc8a6b-c432-11ed-8866-0a147d529a4c,DevicePath:,},},Config:nil,},}
Mar 16 19:40:14.063: INFO: 
Logging kubelet events for node i-0b5975e41089e600d
Mar 16 19:40:14.127: INFO: 
Logging pods the kubelet thinks is on node i-0b5975e41089e600d
Mar 16 19:40:14.204: INFO: execpoddwfvp started at 2023-03-16 19:39:28 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:14.204: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 16 19:40:14.204: INFO: pvc-volume-tester-gbcc6 started at 2023-03-16 19:40:06 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:14.204: INFO: 	Container volume-tester ready: true, restart count 0
Mar 16 19:40:14.204: INFO: csi-mockplugin-0 started at 2023-03-16 19:39:57 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:40:14.204: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:40:14.204: INFO: 	Container driver-registrar ready: true, restart count 0
Mar 16 19:40:14.204: INFO: 	Container mock ready: true, restart count 0
Mar 16 19:40:14.204: INFO: execpod-affinitykj4rg started at 2023-03-16 19:40:10 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:14.204: INFO: 	Container agnhost-container ready: true, restart count 0
Mar 16 19:40:14.204: INFO: affinity-clusterip-transition-49smt started at 2023-03-16 19:40:07 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:14.204: INFO: 	Container affinity-clusterip-transition ready: true, restart count 0
Mar 16 19:40:14.204: INFO: cilium-9vmm6 started at 2023-03-16 19:32:54 +0000 UTC (1+1 container statuses recorded)
Mar 16 19:40:14.204: INFO: 	Init container clean-cilium-state ready: true, restart count 0
Mar 16 19:40:14.204: INFO: 	Container cilium-agent ready: true, restart count 0
Mar 16 19:40:14.204: INFO: csi-mockplugin-attacher-0 started at 2023-03-16 19:39:57 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:14.204: INFO: 	Container csi-attacher ready: true, restart count 0
Mar 16 19:40:14.204: INFO: inline-volume-857wj started at 2023-03-16 19:39:33 +0000 UTC (0+1 container statuses recorded)
Mar 16 19:40:14.204: INFO: 	Container volume-tester ready: false, restart count 0
Mar 16 19:40:14.204: INFO: ebs-csi-node-rkwb9 started at 2023-03-16 19:32:54 +0000 UTC (0+3 container statuses recorded)
Mar 16 19:40:14.204: INFO: 	Container ebs-plugin ready: true, restart count 0
Mar 16 19:40:14.204: INFO: 	Container liveness-probe ready: true, restart count 0
Mar 16 19:40:14.204: INFO: 	Container node-driver-registrar ready: true, restart count 0
Mar 16 19:40:14.204: INFO: csi-mockplugin-0 started at 2023-03-16 19:39:45 +0000 UTC (0+4 container statuses recorded)
Mar 16 19:40:14.204: INFO: 	Container busybox ready: true, restart count 0
Mar 16 19:40:14.204: INFO: 	Container csi-provisioner ready: true, restart count 0
Mar 16 19:40:14.204: INFO: 	Container driver-registrar ready: true, restart count 0
Mar 16 19:40:14.204: INFO: 	Container mock ready: true, restart count 0
Mar 16 19:40:14.530: INFO: 
Latency metrics for node i-0b5975e41089e600d
END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/16/23 19:40:14.53 (3.124s)
< Exit [DeferCleanup (Each)] [sig-network] Services - dump namespaces | framework.go:209 @ 03/16/23 19:40:14.53 (3.124s)
> Enter [DeferCleanup (Each)] [sig-network] Services - tear down framework | framework.go:206 @ 03/16/23 19:40:14.53
STEP: Destroying namespace "services-2463" for this suite. - test/e2e/framework/framework.go:351 @ 03/16/23 19:40:14.53
< Exit [DeferCleanup (Each)] [sig-network] Services - tear down framework | framework.go:206 @ 03/16/23 19:40:14.595 (65ms)
> Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/16/23 19:40:14.595
< Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/16/23 19:40:14.595 (0s)

				Click to see stdout/stderrfrom junit_01.xml

Filter through log files | View test history on testgrid

kubetest2 Test 18m46s

exit status 255
				from junit_runner.xml

Filter through log files | View test history on testgrid

Show 905 Passed Tests

Show 6632 Skipped Tests

Result	FAILURE
Tests	4 failed / 905 succeeded
Started	2023-03-16 19:23
Elapsed	34m51s
Revision	master