Result | FAILURE |
Tests | 94 failed / 675 succeeded |
Started | |
Elapsed | 1h1m |
Revision | master |
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(block\svolmode\)\(allowExpansion\)\]\svolume\-expand\sVerify\sif\soffline\sPVC\sexpansion\sworks$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:174 Dec 2 00:01:03.350: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc00527a740>: { s: "pod \"pod-bdab78f7-ba7b-45c6-85c4-8716bc4cccaf\" is not Running: timed out waiting for the condition", } pod "pod-bdab78f7-ba7b-45c6-85c4-8716bc4cccaf" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:192from junit_14.xml
[BeforeEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Dec 1 23:56:01.195: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename volume-expand �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] Verify if offline PVC expansion works /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:174 Dec 1 23:56:02.082: INFO: Creating resource for dynamic PV Dec 1 23:56:02.082: INFO: Using claimSize:1Gi, test suite supported size:{ 1Gi}, driver(ebs.csi.aws.com) supported size:{ 1Gi} �[1mSTEP�[0m: creating a StorageClass volume-expand-3755-e2e-sc4wgh2 �[1mSTEP�[0m: creating a claim �[1mSTEP�[0m: Creating a pod with dynamically provisioned volume Dec 2 00:01:03.350: FAIL: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc00527a740>: { s: "pod \"pod-bdab78f7-ba7b-45c6-85c4-8716bc4cccaf\" is not Running: timed out waiting for the condition", } pod "pod-bdab78f7-ba7b-45c6-85c4-8716bc4cccaf" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func4() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:192 +0x34d k8s.io/kubernetes/test/e2e.RunE2ETests(0xc0002a6780) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc0002a6780) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc0002a6780, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 Dec 2 00:01:03.350: INFO: Deleting pod "pod-bdab78f7-ba7b-45c6-85c4-8716bc4cccaf" in namespace "volume-expand-3755" Dec 2 00:01:03.532: INFO: Wait up to 5m0s for pod "pod-bdab78f7-ba7b-45c6-85c4-8716bc4cccaf" to be fully deleted �[1mSTEP�[0m: Deleting pod Dec 2 00:01:03.708: INFO: Deleting pod "pod-bdab78f7-ba7b-45c6-85c4-8716bc4cccaf" in namespace "volume-expand-3755" �[1mSTEP�[0m: Deleting pvc Dec 2 00:01:04.060: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comxp98h" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "volume-expand-3755". �[1mSTEP�[0m: Found 5 events. Dec 2 00:01:04.592: INFO: At 2022-12-01 23:56:02 +0000 UTC - event for ebs.csi.aws.comxp98h: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Dec 2 00:01:04.592: INFO: At 2022-12-01 23:56:02 +0000 UTC - event for ebs.csi.aws.comxp98h: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Dec 2 00:01:04.592: INFO: At 2022-12-01 23:56:02 +0000 UTC - event for ebs.csi.aws.comxp98h: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-3755/ebs.csi.aws.comxp98h" Dec 2 00:01:04.592: INFO: At 2022-12-01 23:56:12 +0000 UTC - event for ebs.csi.aws.comxp98h: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-3755-e2e-sc4wgh2": rpc error: code = DeadlineExceeded desc = context deadline exceeded Dec 2 00:01:04.592: INFO: At 2022-12-02 00:01:03 +0000 UTC - event for pod-bdab78f7-ba7b-45c6-85c4-8716bc4cccaf: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "pod-bdab78f7-ba7b-45c6-85c4-8716bc4cccaf" not found Dec 2 00:01:04.768: INFO: POD NODE PHASE GRACE CONDITIONS Dec 2 00:01:04.768: INFO: Dec 2 00:01:04.946: INFO: Logging node info for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:01:05.123: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-156.ap-northeast-2.compute.internal e92fe5b6-ec1b-421a-992a-1182ff1e3522 48951 0 2022-12-01 23:15:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-156.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03fe1aa2fac33129f"} io.cilium.network.ipv4-cilium-host:172.20.40.211 io.cilium.network.ipv4-health-ip:172.20.46.196 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-12-01 23:15:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-12-01 23:15:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-12-01 23:15:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:16:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:21 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03fe1aa2fac33129f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:21 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:16:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.156,},NodeAddress{Type:ExternalIP,Address:15.165.75.247,},NodeAddress{Type:Hostname,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-165-75-247.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccbd8e0eb399c0115bc276d51c7ab,SystemUUID:ec2ccbd8-e0eb-399c-0115-bc276d51c7ab,BootID:4c45c6e5-af61-46fe-b037-b7cc919a53fc,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.2],SizeBytes:41958567,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.2],SizeBytes:41441104,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.2],SizeBytes:4954598,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:01:05.123: INFO: Logging kubelet events for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:01:05.302: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:01:05.490: INFO: ebs-csi-controller-764b64d679-5qsrv started at 2022-12-01 23:15:46 +0000 UTC (0+5 container statuses recorded) Dec 2 00:01:05.490: INFO: Container csi-attacher ready: true, restart count 0 Dec 2 00:01:05.490: INFO: Container csi-provisioner ready: true, restart count 0 Dec 2 00:01:05.490: INFO: Container csi-resizer ready: true, restart count 0 Dec 2 00:01:05.490: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:01:05.490: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:01:05.490: INFO: kube-controller-manager-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:05.490: INFO: Container kube-controller-manager ready: true, restart count 2 Dec 2 00:01:05.490: INFO: etcd-manager-events-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:05.490: INFO: Container etcd-manager ready: true, restart count 0 Dec 2 00:01:05.490: INFO: etcd-manager-main-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:05.490: INFO: Container etcd-manager ready: true, restart count 0 Dec 2 00:01:05.490: INFO: kops-controller-g4pqg started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:05.490: INFO: Container kops-controller ready: true, restart count 0 Dec 2 00:01:05.490: INFO: cilium-pb6vg started at 2022-12-01 23:15:46 +0000 UTC (1+1 container statuses recorded) Dec 2 00:01:05.490: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:01:05.490: INFO: Container cilium-agent ready: true, restart count 1 Dec 2 00:01:05.490: INFO: kube-scheduler-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:05.490: INFO: Container kube-scheduler ready: true, restart count 0 Dec 2 00:01:05.490: INFO: kube-apiserver-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+2 container statuses recorded) Dec 2 00:01:05.490: INFO: Container healthcheck ready: true, restart count 0 Dec 2 00:01:05.490: INFO: Container kube-apiserver ready: true, restart count 1 Dec 2 00:01:05.490: INFO: ebs-csi-node-m7cgg started at 2022-12-01 23:15:46 +0000 UTC (0+3 container statuses recorded) Dec 2 00:01:05.490: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:01:05.490: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:01:05.490: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:01:05.490: INFO: dns-controller-5fb778d9d5-qrfkv started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:05.490: INFO: Container dns-controller ready: true, restart count 0 Dec 2 00:01:05.490: INFO: cilium-operator-587dd66f87-pgs67 started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:05.490: INFO: Container cilium-operator ready: true, restart count 0 Dec 2 00:01:06.085: INFO: Latency metrics for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:01:06.085: INFO: Logging node info for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:01:06.262: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-46-84.ap-northeast-2.compute.internal e408f2b8-c522-4e88-973a-78a2b7a2298f 48987 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-46-84.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-3465 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-46-84.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03a01a80f538d5d6b"} io.cilium.network.ipv4-cilium-host:172.20.38.72 io.cilium.network.ipv4-health-ip:172.20.48.223 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:40 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:40:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kubelet Update v1 2022-12-01 23:57:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03a01a80f538d5d6b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:40 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:57:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.46.84,},NodeAddress{Type:ExternalIP,Address:54.180.162.145,},NodeAddress{Type:Hostname,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-162-145.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec285e4d0d3397ed7e702d489e42654b,SystemUUID:ec285e4d-0d33-97ed-7e70-2d489e42654b,BootID:d6d1d15f-8121-4e40-b71c-e5b51419dc1b,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:01:06.263: INFO: Logging kubelet events for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:01:06.467: INFO: Logging pods the kubelet thinks is on node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:01:06.646: INFO: cilium-7h664 started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 2 00:01:06.646: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:01:06.646: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:01:06.646: INFO: ebs-csi-node-lfxzl started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 2 00:01:06.646: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:01:06.646: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:01:06.646: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:01:07.233: INFO: Latency metrics for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:01:07.234: INFO: Logging node info for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:01:07.410: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-54-150.ap-northeast-2.compute.internal 2a8a1d71-d00b-4259-82bd-841f96508593 48413 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-54-150.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-54-150.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02d689db76221fe72"} io.cilium.network.ipv4-cilium-host:172.20.43.0 io.cilium.network.ipv4-health-ip:172.20.58.253 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-12-01 23:17:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:38 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:56:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02d689db76221fe72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:38 +0000 UTC,LastTransitionTime:2022-12-01 23:17:38 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:57:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:30 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.150,},NodeAddress{Type:ExternalIP,Address:54.180.133.96,},NodeAddress{Type:Hostname,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-133-96.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec20870a7ff0c6740650f49249abc430,SystemUUID:ec20870a-7ff0-c674-0650-f49249abc430,BootID:803a40bf-5805-4ae9-b933-fde612d75f1d,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 k8s.gcr.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:e7fddbaac4c3451da2365ab90bad149d32f11409738034e41e0f460927f7c276 k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.4],SizeBytes:24757245,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:01:07.411: INFO: Logging kubelet events for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:01:07.589: INFO: Logging pods the kubelet thinks is on node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:01:07.768: INFO: ebs-csi-node-n5tcr started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 2 00:01:07.768: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:01:07.768: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:01:07.768: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:01:07.768: INFO: cilium-2bxvd started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 2 00:01:07.768: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:01:07.768: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:01:07.768: INFO: coredns-5fcc7b6498-vcsml started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:07.768: INFO: Container coredns ready: true, restart count 0 Dec 2 00:01:07.768: INFO: coredns-autoscaler-6658b4bf85-92nhw started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:07.768: INFO: Container autoscaler ready: true, restart count 0 Dec 2 00:01:08.365: INFO: Latency metrics for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:01:08.365: INFO: Logging node info for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:01:08.541: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-57.ap-northeast-2.compute.internal 2ea0d3e0-dcd9-4562-ada8-ee5e6b866480 49228 0 2022-12-01 23:17:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-57.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-55-57.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02200981456309a72"} io.cilium.network.ipv4-cilium-host:172.20.51.99 io.cilium.network.ipv4-health-ip:172.20.36.120 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:44 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:54:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02200981456309a72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:44 +0000 UTC,LastTransitionTime:2022-12-01 23:17:44 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:57:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.57,},NodeAddress{Type:ExternalIP,Address:3.36.130.97,},NodeAddress{Type:Hostname,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-36-130-97.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bf2fd9fd22cd1aac5da0aa67f7a08,SystemUUID:ec2bf2fd-9fd2-2cd1-aac5-da0aa67f7a08,BootID:c4c37f48-1b62-417d-82bf-ea460350f712,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-936^47b47419-71d1-11ed-be23-02bdc8e24a07],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:01:08.543: INFO: Logging kubelet events for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:01:08.722: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:01:08.902: INFO: ebs-csi-node-5zdr7 started at 2022-12-01 23:17:15 +0000 UTC (0+3 container statuses recorded) Dec 2 00:01:08.902: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:01:08.902: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:01:08.902: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:01:08.902: INFO: cilium-nwg58 started at 2022-12-01 23:17:15 +0000 UTC (1+1 container statuses recorded) Dec 2 00:01:08.902: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:01:08.902: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:01:08.902: INFO: coredns-5fcc7b6498-x9kjd started at 2022-12-01 23:17:56 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:08.902: INFO: Container coredns ready: true, restart count 0 Dec 2 00:01:09.522: INFO: Latency metrics for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:01:09.522: INFO: Logging node info for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:01:09.699: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-60-229.ap-northeast-2.compute.internal 63277852-d67d-402c-b963-ae3a2995d791 49657 0 2022-12-01 23:17:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-60-229.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-60-229.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0441b4e9d8ac12445"} io.cilium.network.ipv4-cilium-host:172.20.50.202 io.cilium.network.ipv4-health-ip:172.20.42.191 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:45 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:59:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-0441b4e9d8ac12445,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:45 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.229,},NodeAddress{Type:ExternalIP,Address:43.201.70.39,},NodeAddress{Type:Hostname,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-43-201-70-39.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec24f7651bea0c9073fca065a65b3820,SystemUUID:ec24f765-1bea-0c90-73fc-a065a65b3820,BootID:5386d7df-0b64-4c56-85ff-3f8a9a84c900,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:01:09.700: INFO: Logging kubelet events for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:01:09.878: INFO: Logging pods the kubelet thinks is on node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:01:10.057: INFO: ebs-csi-node-wqh44 started at 2022-12-01 23:17:12 +0000 UTC (0+3 container statuses recorded) Dec 2 00:01:10.057: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:01:10.057: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:01:10.057: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:01:10.057: INFO: cilium-5dnwd started at 2022-12-01 23:17:12 +0000 UTC (1+1 container statuses recorded) Dec 2 00:01:10.057: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:01:10.057: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:01:10.649: INFO: Latency metrics for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:01:10.649: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volume-expand-3755" for this suite.
Find pod-bdab78f7-ba7b-45c6-85c4-8716bc4cccaf mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(block\svolmode\)\(allowExpansion\)\]\svolume\-expand\sshould\sresize\svolume\swhen\sPVC\sis\sedited\swhile\spod\sis\susing\sit$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:246 Dec 1 23:47:21.163: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc004c00160>: { s: "pod \"pod-4ea5de8f-2678-40cd-9857-6dd7bbde80f5\" is not Running: timed out waiting for the condition", } pod "pod-4ea5de8f-2678-40cd-9857-6dd7bbde80f5" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:268from junit_11.xml
[BeforeEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Dec 1 23:42:19.030: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename volume-expand �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should resize volume when PVC is edited while pod is using it /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:246 Dec 1 23:42:19.919: INFO: Creating resource for dynamic PV Dec 1 23:42:19.919: INFO: Using claimSize:1Gi, test suite supported size:{ 1Gi}, driver(ebs.csi.aws.com) supported size:{ 1Gi} �[1mSTEP�[0m: creating a StorageClass volume-expand-9799-e2e-sc7mr7d �[1mSTEP�[0m: creating a claim �[1mSTEP�[0m: Creating a pod with dynamically provisioned volume Dec 1 23:47:21.162: FAIL: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc004c00160>: { s: "pod \"pod-4ea5de8f-2678-40cd-9857-6dd7bbde80f5\" is not Running: timed out waiting for the condition", } pod "pod-4ea5de8f-2678-40cd-9857-6dd7bbde80f5" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func5() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:268 +0x3d0 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000726000) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000726000) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000726000, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 Dec 1 23:47:21.163: INFO: Deleting pod "pod-4ea5de8f-2678-40cd-9857-6dd7bbde80f5" in namespace "volume-expand-9799" Dec 1 23:47:21.344: INFO: Wait up to 5m0s for pod "pod-4ea5de8f-2678-40cd-9857-6dd7bbde80f5" to be fully deleted �[1mSTEP�[0m: Deleting pod Dec 1 23:47:21.526: INFO: Deleting pod "pod-4ea5de8f-2678-40cd-9857-6dd7bbde80f5" in namespace "volume-expand-9799" �[1mSTEP�[0m: Deleting pvc Dec 1 23:47:21.880: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comlkqjx" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "volume-expand-9799". �[1mSTEP�[0m: Found 6 events. Dec 1 23:47:22.416: INFO: At 2022-12-01 23:42:20 +0000 UTC - event for ebs.csi.aws.comlkqjx: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Dec 1 23:47:22.416: INFO: At 2022-12-01 23:42:20 +0000 UTC - event for ebs.csi.aws.comlkqjx: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Dec 1 23:47:22.416: INFO: At 2022-12-01 23:42:20 +0000 UTC - event for ebs.csi.aws.comlkqjx: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-9799/ebs.csi.aws.comlkqjx" Dec 1 23:47:22.416: INFO: At 2022-12-01 23:42:30 +0000 UTC - event for ebs.csi.aws.comlkqjx: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-9799-e2e-sc7mr7d": rpc error: code = DeadlineExceeded desc = context deadline exceeded Dec 1 23:47:22.416: INFO: At 2022-12-01 23:42:53 +0000 UTC - event for ebs.csi.aws.comlkqjx: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-9799-e2e-sc7mr7d": rpc error: code = Internal desc = Could not create volume "pvc-3f1f1748-2d5a-46d4-9160-a6a33d68b925": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Dec 1 23:47:22.416: INFO: At 2022-12-01 23:47:21 +0000 UTC - event for pod-4ea5de8f-2678-40cd-9857-6dd7bbde80f5: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "pod-4ea5de8f-2678-40cd-9857-6dd7bbde80f5" not found Dec 1 23:47:22.592: INFO: POD NODE PHASE GRACE CONDITIONS Dec 1 23:47:22.592: INFO: Dec 1 23:47:22.773: INFO: Logging node info for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:47:22.966: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-156.ap-northeast-2.compute.internal e92fe5b6-ec1b-421a-992a-1182ff1e3522 22013 0 2022-12-01 23:15:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-156.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03fe1aa2fac33129f"} io.cilium.network.ipv4-cilium-host:172.20.40.211 io.cilium.network.ipv4-health-ip:172.20.46.196 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-12-01 23:15:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-12-01 23:15:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-12-01 23:15:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:16:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:21 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03fe1aa2fac33129f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:21 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:43:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:43:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:43:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:43:02 +0000 UTC,LastTransitionTime:2022-12-01 23:16:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.156,},NodeAddress{Type:ExternalIP,Address:15.165.75.247,},NodeAddress{Type:Hostname,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-165-75-247.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccbd8e0eb399c0115bc276d51c7ab,SystemUUID:ec2ccbd8-e0eb-399c-0115-bc276d51c7ab,BootID:4c45c6e5-af61-46fe-b037-b7cc919a53fc,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.2],SizeBytes:41958567,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.2],SizeBytes:41441104,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.2],SizeBytes:4954598,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:47:22.967: INFO: Logging kubelet events for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:47:23.146: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:47:23.334: INFO: ebs-csi-node-m7cgg started at 2022-12-01 23:15:46 +0000 UTC (0+3 container statuses recorded) Dec 1 23:47:23.334: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:47:23.334: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:47:23.334: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:47:23.334: INFO: dns-controller-5fb778d9d5-qrfkv started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:23.334: INFO: Container dns-controller ready: true, restart count 0 Dec 1 23:47:23.335: INFO: cilium-operator-587dd66f87-pgs67 started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:23.335: INFO: Container cilium-operator ready: true, restart count 0 Dec 1 23:47:23.335: INFO: kube-scheduler-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:23.335: INFO: Container kube-scheduler ready: true, restart count 0 Dec 1 23:47:23.335: INFO: kube-apiserver-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+2 container statuses recorded) Dec 1 23:47:23.335: INFO: Container healthcheck ready: true, restart count 0 Dec 1 23:47:23.335: INFO: Container kube-apiserver ready: true, restart count 1 Dec 1 23:47:23.335: INFO: etcd-manager-main-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:23.335: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:47:23.335: INFO: kops-controller-g4pqg started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:23.335: INFO: Container kops-controller ready: true, restart count 0 Dec 1 23:47:23.335: INFO: cilium-pb6vg started at 2022-12-01 23:15:46 +0000 UTC (1+1 container statuses recorded) Dec 1 23:47:23.335: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:47:23.335: INFO: Container cilium-agent ready: true, restart count 1 Dec 1 23:47:23.335: INFO: ebs-csi-controller-764b64d679-5qsrv started at 2022-12-01 23:15:46 +0000 UTC (0+5 container statuses recorded) Dec 1 23:47:23.335: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:47:23.335: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:47:23.335: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:47:23.335: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:47:23.335: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:47:23.335: INFO: kube-controller-manager-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:23.335: INFO: Container kube-controller-manager ready: true, restart count 2 Dec 1 23:47:23.335: INFO: etcd-manager-events-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:23.335: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:47:24.013: INFO: Latency metrics for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:47:24.013: INFO: Logging node info for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:47:24.192: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-46-84.ap-northeast-2.compute.internal e408f2b8-c522-4e88-973a-78a2b7a2298f 28204 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-46-84.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-3465 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-46-84.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03a01a80f538d5d6b"} io.cilium.network.ipv4-cilium-host:172.20.38.72 io.cilium.network.ipv4-health-ip:172.20.48.223 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:40 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:40:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kubelet Update v1 2022-12-01 23:46:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03a01a80f538d5d6b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:40 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:46:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.46.84,},NodeAddress{Type:ExternalIP,Address:54.180.162.145,},NodeAddress{Type:Hostname,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-162-145.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec285e4d0d3397ed7e702d489e42654b,SystemUUID:ec285e4d-0d33-97ed-7e70-2d489e42654b,BootID:d6d1d15f-8121-4e40-b71c-e5b51419dc1b,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:47:24.193: INFO: Logging kubelet events for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:47:24.381: INFO: Logging pods the kubelet thinks is on node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:47:24.568: INFO: pod-subpath-test-inlinevolume-ptjp started at 2022-12-01 23:47:06 +0000 UTC (1+1 container statuses recorded) Dec 1 23:47:24.569: INFO: Init container init-volume-inlinevolume-ptjp ready: true, restart count 0 Dec 1 23:47:24.569: INFO: Container test-container-subpath-inlinevolume-ptjp ready: true, restart count 0 Dec 1 23:47:24.569: INFO: simpletest-rc-to-be-deleted-jsgkc started at 2022-12-01 23:47:17 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:24.569: INFO: Container nginx ready: true, restart count 0 Dec 1 23:47:24.569: INFO: startup-c27cccc0-443b-48a5-b22e-3d66944d1076 started at 2022-12-01 23:46:56 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:24.569: INFO: Container busybox ready: true, restart count 0 Dec 1 23:47:24.569: INFO: ebs-csi-node-lfxzl started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:47:24.569: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:47:24.569: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:47:24.569: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:47:24.569: INFO: simpletest-rc-to-be-deleted-q69bd started at 2022-12-01 23:47:17 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:24.569: INFO: Container nginx ready: true, restart count 0 Dec 1 23:47:24.569: INFO: pod-c6109d98-d510-485d-8c6b-9a30835cd935 started at 2022-12-01 23:47:18 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:24.569: INFO: Container write-pod ready: true, restart count 0 Dec 1 23:47:24.569: INFO: cilium-7h664 started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:47:24.569: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:47:24.569: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:47:24.569: INFO: startup-714ba7b9-cfb1-40a5-ad4c-f7bfba82598d started at 2022-12-01 23:44:50 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:24.569: INFO: Container busybox ready: false, restart count 0 Dec 1 23:47:24.569: INFO: hostexec-ip-172-20-46-84.ap-northeast-2.compute.internal-wq2x5 started at 2022-12-01 23:47:13 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:24.569: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:47:25.178: INFO: Latency metrics for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:47:25.178: INFO: Logging node info for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:47:25.355: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-54-150.ap-northeast-2.compute.internal 2a8a1d71-d00b-4259-82bd-841f96508593 29523 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-54-150.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-54-150.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02d689db76221fe72"} io.cilium.network.ipv4-cilium-host:172.20.43.0 io.cilium.network.ipv4-health-ip:172.20.58.253 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-12-01 23:17:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:38 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:41:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02d689db76221fe72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:38 +0000 UTC,LastTransitionTime:2022-12-01 23:17:38 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:47:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:47:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:47:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:47:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:30 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.150,},NodeAddress{Type:ExternalIP,Address:54.180.133.96,},NodeAddress{Type:Hostname,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-133-96.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec20870a7ff0c6740650f49249abc430,SystemUUID:ec20870a-7ff0-c674-0650-f49249abc430,BootID:803a40bf-5805-4ae9-b933-fde612d75f1d,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 k8s.gcr.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:e7fddbaac4c3451da2365ab90bad149d32f11409738034e41e0f460927f7c276 k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.4],SizeBytes:24757245,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:47:25.356: INFO: Logging kubelet events for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:47:25.535: INFO: Logging pods the kubelet thinks is on node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:47:25.893: INFO: cilium-2bxvd started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:47:25.893: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:47:25.893: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:47:25.893: INFO: coredns-5fcc7b6498-vcsml started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:25.893: INFO: Container coredns ready: true, restart count 0 Dec 1 23:47:25.893: INFO: coredns-autoscaler-6658b4bf85-92nhw started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:25.893: INFO: Container autoscaler ready: true, restart count 0 Dec 1 23:47:25.893: INFO: hostexec-ip-172-20-54-150.ap-northeast-2.compute.internal-8xhqf started at 2022-12-01 23:47:12 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:25.893: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:47:25.893: INFO: affinity-clusterip-timeout-7s5nr started at 2022-12-01 23:47:15 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:25.893: INFO: Container affinity-clusterip-timeout ready: true, restart count 0 Dec 1 23:47:25.893: INFO: simpletest-rc-to-be-deleted-pkz6l started at 2022-12-01 23:47:17 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:25.893: INFO: Container nginx ready: false, restart count 0 Dec 1 23:47:25.893: INFO: simpletest-rc-to-be-deleted-wt97w started at 2022-12-01 23:47:17 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:25.893: INFO: Container nginx ready: false, restart count 0 Dec 1 23:47:25.893: INFO: simpletest-rc-to-be-deleted-fsf2z started at 2022-12-01 23:47:18 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:25.893: INFO: Container nginx ready: true, restart count 0 Dec 1 23:47:25.893: INFO: ebs-csi-node-n5tcr started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:47:25.893: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:47:25.893: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:47:25.893: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:47:25.893: INFO: pod-sharedvolume-1c24d641-001e-4da3-bb1d-7cf72587a105 started at 2022-12-01 23:47:15 +0000 UTC (0+2 container statuses recorded) Dec 1 23:47:25.893: INFO: Container busybox-main-container ready: true, restart count 0 Dec 1 23:47:25.893: INFO: Container busybox-sub-container ready: false, restart count 0 Dec 1 23:47:26.497: INFO: Latency metrics for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:47:26.497: INFO: Logging node info for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:47:26.675: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-57.ap-northeast-2.compute.internal 2ea0d3e0-dcd9-4562-ada8-ee5e6b866480 29483 0 2022-12-01 23:17:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-57.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-55-57.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02200981456309a72"} io.cilium.network.ipv4-cilium-host:172.20.51.99 io.cilium.network.ipv4-health-ip:172.20.36.120 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:44 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02200981456309a72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:44 +0000 UTC,LastTransitionTime:2022-12-01 23:17:44 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.57,},NodeAddress{Type:ExternalIP,Address:3.36.130.97,},NodeAddress{Type:Hostname,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-36-130-97.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bf2fd9fd22cd1aac5da0aa67f7a08,SystemUUID:ec2bf2fd-9fd2-2cd1-aac5-da0aa67f7a08,BootID:c4c37f48-1b62-417d-82bf-ea460350f712,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-936^47b47419-71d1-11ed-be23-02bdc8e24a07],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:47:26.676: INFO: Logging kubelet events for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:47:26.854: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:47:27.050: INFO: busybox-host-aliasesa272f97f-39ca-4a27-8bf3-0ff90eea2bd8 started at 2022-12-01 23:47:15 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:27.050: INFO: Container busybox-host-aliasesa272f97f-39ca-4a27-8bf3-0ff90eea2bd8 ready: true, restart count 0 Dec 1 23:47:27.050: INFO: ebs-csi-node-5zdr7 started at 2022-12-01 23:17:15 +0000 UTC (0+3 container statuses recorded) Dec 1 23:47:27.050: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:47:27.050: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:47:27.050: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:47:27.050: INFO: simpletest-rc-to-be-deleted-k74pf started at 2022-12-01 23:47:17 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:27.050: INFO: Container nginx ready: true, restart count 0 Dec 1 23:47:27.050: INFO: pod-8fd16826-4031-4a56-9051-24f7a818bd10 started at 2022-12-01 23:47:18 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:27.050: INFO: Container write-pod ready: true, restart count 0 Dec 1 23:47:27.050: INFO: cilium-nwg58 started at 2022-12-01 23:17:15 +0000 UTC (1+1 container statuses recorded) Dec 1 23:47:27.050: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:47:27.050: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:47:27.050: INFO: hostexec-ip-172-20-55-57.ap-northeast-2.compute.internal-jlkfd started at 2022-12-01 23:47:00 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:27.050: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:47:27.050: INFO: affinity-clusterip-timeout-2rjwc started at 2022-12-01 23:47:15 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:27.050: INFO: Container affinity-clusterip-timeout ready: true, restart count 0 Dec 1 23:47:27.050: INFO: coredns-5fcc7b6498-x9kjd started at 2022-12-01 23:17:56 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:27.050: INFO: Container coredns ready: true, restart count 0 Dec 1 23:47:27.050: INFO: hostexec-ip-172-20-55-57.ap-northeast-2.compute.internal-nsbgj started at 2022-12-01 23:47:23 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:27.050: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:47:27.050: INFO: simpletest-rc-to-be-deleted-cj5kj started at 2022-12-01 23:47:18 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:27.050: INFO: Container nginx ready: true, restart count 0 Dec 1 23:47:27.687: INFO: Latency metrics for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:47:27.687: INFO: Logging node info for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:47:27.868: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-60-229.ap-northeast-2.compute.internal 63277852-d67d-402c-b963-ae3a2995d791 29243 0 2022-12-01 23:17:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-60-229.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-60-229.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0441b4e9d8ac12445"} io.cilium.network.ipv4-cilium-host:172.20.50.202 io.cilium.network.ipv4-health-ip:172.20.42.191 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:45 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:41:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-0441b4e9d8ac12445,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:45 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:46:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.229,},NodeAddress{Type:ExternalIP,Address:43.201.70.39,},NodeAddress{Type:Hostname,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-43-201-70-39.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec24f7651bea0c9073fca065a65b3820,SystemUUID:ec24f765-1bea-0c90-73fc-a065a65b3820,BootID:5386d7df-0b64-4c56-85ff-3f8a9a84c900,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:47:27.869: INFO: Logging kubelet events for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:47:28.049: INFO: Logging pods the kubelet thinks is on node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:47:28.231: INFO: execpod-affinityqhbdj started at 2022-12-01 23:47:22 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:28.231: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:47:28.231: INFO: ebs-csi-node-wqh44 started at 2022-12-01 23:17:12 +0000 UTC (0+3 container statuses recorded) Dec 1 23:47:28.231: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:47:28.231: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:47:28.231: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:47:28.231: INFO: cilium-5dnwd started at 2022-12-01 23:17:12 +0000 UTC (1+1 container statuses recorded) Dec 1 23:47:28.231: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:47:28.231: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:47:28.231: INFO: svc-latency-rc-7lqrq started at 2022-12-01 23:47:19 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:28.231: INFO: Container svc-latency-rc ready: true, restart count 0 Dec 1 23:47:28.231: INFO: simpletest-rc-to-be-deleted-4qvds started at 2022-12-01 23:47:17 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:28.231: INFO: Container nginx ready: true, restart count 0 Dec 1 23:47:28.231: INFO: affinity-clusterip-timeout-ts4fn started at 2022-12-01 23:47:15 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:28.231: INFO: Container affinity-clusterip-timeout ready: true, restart count 0 Dec 1 23:47:28.231: INFO: service-proxy-disabled-9ltgd started at 2022-12-01 23:47:27 +0000 UTC (0+1 container statuses recorded) Dec 1 23:47:28.231: INFO: Container service-proxy-disabled ready: false, restart count 0 Dec 1 23:47:28.900: INFO: Latency metrics for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:47:28.900: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volume-expand-9799" for this suite.
Find pod-4ea5de8f-2678-40cd-9857-6dd7bbde80f5 mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(block\svolmode\)\]\svolumeMode\sshould\snot\smount\s\/\smap\sunused\svolumes\sin\sa\spod\s\[LinuxOnly\]$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:352 Dec 1 23:25:48.831: Unexpected error: <*errors.errorString | 0xc000248290>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:385from junit_16.xml
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Dec 1 23:20:46.514: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename volumemode W1201 23:20:47.226028 6655 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ Dec 1 23:20:47.226: INFO: No PodSecurityPolicies found; assuming PodSecurityPolicy is disabled. �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should not mount / map unused volumes in a pod [LinuxOnly] /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:352 Dec 1 23:20:47.578: INFO: Creating resource for dynamic PV Dec 1 23:20:47.578: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass volumemode-8642-e2e-scc6wxj �[1mSTEP�[0m: creating a claim �[1mSTEP�[0m: Creating pod Dec 1 23:25:48.830: FAIL: Unexpected error: <*errors.errorString | 0xc000248290>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeModeTestSuite).DefineTests.func7() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:385 +0x6c5 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000227380) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000227380) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000227380, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 Dec 1 23:25:48.831: INFO: Deleting pod "pod-268cc99a-d73d-4bcc-af87-363fab100140" in namespace "volumemode-8642" Dec 1 23:25:49.013: INFO: Wait up to 5m0s for pod "pod-268cc99a-d73d-4bcc-af87-363fab100140" to be fully deleted �[1mSTEP�[0m: Deleting pvc Dec 1 23:25:49.366: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comg4kxc" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (block volmode)] volumeMode /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "volumemode-8642". �[1mSTEP�[0m: Found 5 events. Dec 1 23:25:49.900: INFO: At 2022-12-01 23:20:47 +0000 UTC - event for ebs.csi.aws.comg4kxc: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Dec 1 23:25:49.900: INFO: At 2022-12-01 23:20:48 +0000 UTC - event for ebs.csi.aws.comg4kxc: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } Provisioning: External provisioner is provisioning volume for claim "volumemode-8642/ebs.csi.aws.comg4kxc" Dec 1 23:25:49.900: INFO: At 2022-12-01 23:20:48 +0000 UTC - event for ebs.csi.aws.comg4kxc: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Dec 1 23:25:49.900: INFO: At 2022-12-01 23:20:58 +0000 UTC - event for ebs.csi.aws.comg4kxc: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "volumemode-8642-e2e-scc6wxj": rpc error: code = DeadlineExceeded desc = context deadline exceeded Dec 1 23:25:49.900: INFO: At 2022-12-01 23:25:49 +0000 UTC - event for pod-268cc99a-d73d-4bcc-af87-363fab100140: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "pod-268cc99a-d73d-4bcc-af87-363fab100140" not found Dec 1 23:25:50.077: INFO: POD NODE PHASE GRACE CONDITIONS Dec 1 23:25:50.077: INFO: Dec 1 23:25:50.433: INFO: Logging node info for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:25:50.610: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-156.ap-northeast-2.compute.internal e92fe5b6-ec1b-421a-992a-1182ff1e3522 5601 0 2022-12-01 23:15:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-156.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03fe1aa2fac33129f"} io.cilium.network.ipv4-cilium-host:172.20.40.211 io.cilium.network.ipv4-health-ip:172.20.46.196 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-12-01 23:15:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-12-01 23:15:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-12-01 23:15:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:16:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:21 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03fe1aa2fac33129f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:21 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:23:01 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:23:01 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:23:01 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:23:01 +0000 UTC,LastTransitionTime:2022-12-01 23:16:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.156,},NodeAddress{Type:ExternalIP,Address:15.165.75.247,},NodeAddress{Type:Hostname,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-165-75-247.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccbd8e0eb399c0115bc276d51c7ab,SystemUUID:ec2ccbd8-e0eb-399c-0115-bc276d51c7ab,BootID:4c45c6e5-af61-46fe-b037-b7cc919a53fc,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.2],SizeBytes:41958567,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.2],SizeBytes:41441104,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.2],SizeBytes:4954598,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:25:50.611: INFO: Logging kubelet events for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:25:50.790: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:25:51.155: INFO: kube-controller-manager-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:51.155: INFO: Container kube-controller-manager ready: true, restart count 2 Dec 1 23:25:51.155: INFO: etcd-manager-events-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:51.155: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:25:51.155: INFO: etcd-manager-main-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:51.155: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:25:51.155: INFO: kops-controller-g4pqg started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:51.155: INFO: Container kops-controller ready: true, restart count 0 Dec 1 23:25:51.155: INFO: cilium-pb6vg started at 2022-12-01 23:15:46 +0000 UTC (1+1 container statuses recorded) Dec 1 23:25:51.155: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:25:51.155: INFO: Container cilium-agent ready: true, restart count 1 Dec 1 23:25:51.155: INFO: ebs-csi-controller-764b64d679-5qsrv started at 2022-12-01 23:15:46 +0000 UTC (0+5 container statuses recorded) Dec 1 23:25:51.155: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:25:51.155: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:25:51.155: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:25:51.155: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:25:51.155: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:25:51.155: INFO: kube-scheduler-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:51.155: INFO: Container kube-scheduler ready: true, restart count 0 Dec 1 23:25:51.155: INFO: kube-apiserver-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+2 container statuses recorded) Dec 1 23:25:51.155: INFO: Container healthcheck ready: true, restart count 0 Dec 1 23:25:51.155: INFO: Container kube-apiserver ready: true, restart count 1 Dec 1 23:25:51.155: INFO: ebs-csi-node-m7cgg started at 2022-12-01 23:15:46 +0000 UTC (0+3 container statuses recorded) Dec 1 23:25:51.155: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:25:51.155: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:25:51.155: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:25:51.155: INFO: dns-controller-5fb778d9d5-qrfkv started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:51.155: INFO: Container dns-controller ready: true, restart count 0 Dec 1 23:25:51.155: INFO: cilium-operator-587dd66f87-pgs67 started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:51.155: INFO: Container cilium-operator ready: true, restart count 0 Dec 1 23:25:51.768: INFO: Latency metrics for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:25:51.768: INFO: Logging node info for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:25:51.945: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-46-84.ap-northeast-2.compute.internal e408f2b8-c522-4e88-973a-78a2b7a2298f 5114 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-46-84.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03a01a80f538d5d6b"} io.cilium.network.ipv4-cilium-host:172.20.38.72 io.cilium.network.ipv4-health-ip:172.20.48.223 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:40 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:18:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {e2e.test Update v1 2022-12-01 23:21:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03a01a80f538d5d6b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:40 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.46.84,},NodeAddress{Type:ExternalIP,Address:54.180.162.145,},NodeAddress{Type:Hostname,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-162-145.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec285e4d0d3397ed7e702d489e42654b,SystemUUID:ec285e4d-0d33-97ed-7e70-2d489e42654b,BootID:d6d1d15f-8121-4e40-b71c-e5b51419dc1b,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:25:51.946: INFO: Logging kubelet events for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:25:52.125: INFO: Logging pods the kubelet thinks is on node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:25:52.309: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-stdpc started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:52.309: INFO: affinity-clusterip-transition-4bl6d started at 2022-12-01 23:21:24 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container affinity-clusterip-transition ready: true, restart count 0 Dec 1 23:25:52.309: INFO: cilium-7h664 started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:25:52.309: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:25:52.309: INFO: hostexec-ip-172-20-46-84.ap-northeast-2.compute.internal-5rqtf started at 2022-12-01 23:23:40 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:25:52.309: INFO: netserver-0 started at 2022-12-01 23:22:07 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container webserver ready: true, restart count 0 Dec 1 23:25:52.309: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-lwxkw started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:25:52.309: INFO: netserver-0 started at 2022-12-01 23:22:25 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container webserver ready: true, restart count 0 Dec 1 23:25:52.309: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-wmskh started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:52.309: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-m9cn6 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:52.309: INFO: webserver-deployment-847dcfb7fb-ng82l started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container httpd ready: false, restart count 0 Dec 1 23:25:52.309: INFO: hostexec-ip-172-20-46-84.ap-northeast-2.compute.internal-zx922 started at 2022-12-01 23:21:19 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:25:52.309: INFO: webserver-deployment-847dcfb7fb-x4hj5 started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container httpd ready: false, restart count 0 Dec 1 23:25:52.309: INFO: pod-c65bf730-a2f5-41c7-b67d-7e37f8ac5796 started at 2022-12-01 23:22:32 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container write-pod ready: false, restart count 0 Dec 1 23:25:52.309: INFO: pod-subpath-test-preprovisionedpv-pr5s started at 2022-12-01 23:23:47 +0000 UTC (0+2 container statuses recorded) Dec 1 23:25:52.309: INFO: Container test-container-subpath-preprovisionedpv-pr5s ready: false, restart count 0 Dec 1 23:25:52.309: INFO: Container test-container-volume-preprovisionedpv-pr5s ready: false, restart count 0 Dec 1 23:25:52.309: INFO: webserver-deployment-847dcfb7fb-hch2v started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container httpd ready: false, restart count 0 Dec 1 23:25:52.309: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-4t8x8 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:52.309: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-zdrcp started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:52.309: INFO: netserver-0 started at 2022-12-01 23:21:41 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container webserver ready: true, restart count 0 Dec 1 23:25:52.309: INFO: pod-subpath-test-preprovisionedpv-crsm started at 2022-12-01 23:21:48 +0000 UTC (2+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Init container init-volume-preprovisionedpv-crsm ready: false, restart count 0 Dec 1 23:25:52.309: INFO: Init container test-init-subpath-preprovisionedpv-crsm ready: false, restart count 0 Dec 1 23:25:52.309: INFO: Container test-container-subpath-preprovisionedpv-crsm ready: false, restart count 0 Dec 1 23:25:52.309: INFO: hostexec-ip-172-20-46-84.ap-northeast-2.compute.internal-b2j6s started at 2022-12-01 23:22:14 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:25:52.309: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-6rhs5 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:52.309: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-2k8kk started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:25:52.309: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-kdzjv started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:52.309: INFO: ebs-csi-node-lfxzl started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:25:52.309: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:25:52.309: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:25:52.309: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:25:52.309: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-gqqcm started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:52.309: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:52.917: INFO: Latency metrics for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:25:52.917: INFO: Logging node info for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:25:53.094: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-54-150.ap-northeast-2.compute.internal 2a8a1d71-d00b-4259-82bd-841f96508593 5111 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-54-150.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-54-150.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-3080":"ip-172-20-54-150.ap-northeast-2.compute.internal","ebs.csi.aws.com":"i-02d689db76221fe72"} io.cilium.network.ipv4-cilium-host:172.20.43.0 io.cilium.network.ipv4-health-ip:172.20.58.253 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-12-01 23:17:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:38 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:21:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:22:00 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-01 23:22:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02d689db76221fe72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:38 +0000 UTC,LastTransitionTime:2022-12-01 23:17:38 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:30 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.150,},NodeAddress{Type:ExternalIP,Address:54.180.133.96,},NodeAddress{Type:Hostname,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-133-96.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec20870a7ff0c6740650f49249abc430,SystemUUID:ec20870a-7ff0-c674-0650-f49249abc430,BootID:803a40bf-5805-4ae9-b933-fde612d75f1d,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-3080^f43ea0ef-71ce-11ed-92c1-0628feb04e56 kubernetes.io/csi/csi-hostpath-ephemeral-3080^f800f595-71ce-11ed-92c1-0628feb04e56],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-3080^f43ea0ef-71ce-11ed-92c1-0628feb04e56,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-3080^f800f595-71ce-11ed-92c1-0628feb04e56,DevicePath:,},},Config:nil,},} Dec 1 23:25:53.095: INFO: Logging kubelet events for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:25:53.274: INFO: Logging pods the kubelet thinks is on node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:25:53.460: INFO: inline-volume-tester-zzzxb started at 2022-12-01 23:21:59 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container csi-volume-tester ready: true, restart count 0 Dec 1 23:25:53.460: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-j8c74 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:53.460: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-7qcv7 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:53.460: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-6jcw4 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:25:53.460: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-htn5g started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:25:53.460: INFO: netserver-1 started at 2022-12-01 23:22:07 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container webserver ready: false, restart count 0 Dec 1 23:25:53.460: INFO: coredns-autoscaler-6658b4bf85-92nhw started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container autoscaler ready: true, restart count 0 Dec 1 23:25:53.460: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-5d7kh started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:53.460: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-6nhjl started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:53.460: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-2kzlf started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:53.460: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-kq9dz started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:53.460: INFO: webserver-deployment-847dcfb7fb-p9cpm started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container httpd ready: false, restart count 0 Dec 1 23:25:53.460: INFO: webserver-deployment-847dcfb7fb-qb48w started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container httpd ready: false, restart count 0 Dec 1 23:25:53.460: INFO: ebs-csi-node-n5tcr started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:25:53.460: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:25:53.460: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:25:53.460: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:25:53.460: INFO: coredns-5fcc7b6498-vcsml started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container coredns ready: true, restart count 0 Dec 1 23:25:53.460: INFO: inline-volume-tester2-86kbx started at 2022-12-01 23:22:06 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container csi-volume-tester ready: false, restart count 0 Dec 1 23:25:53.460: INFO: netserver-1 started at 2022-12-01 23:22:25 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.460: INFO: Container webserver ready: false, restart count 0 Dec 1 23:25:53.460: INFO: netserver-1 started at 2022-12-01 23:21:41 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.461: INFO: Container webserver ready: true, restart count 0 Dec 1 23:25:53.461: INFO: csi-hostpathplugin-0 started at 2022-12-01 23:21:42 +0000 UTC (0+7 container statuses recorded) Dec 1 23:25:53.461: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:25:53.461: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:25:53.461: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:25:53.461: INFO: Container csi-snapshotter ready: true, restart count 0 Dec 1 23:25:53.461: INFO: Container hostpath ready: true, restart count 0 Dec 1 23:25:53.461: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:25:53.461: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:25:53.461: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-r5sld started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.461: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:53.461: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-5xvz5 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:53.461: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:25:53.461: INFO: cilium-2bxvd started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:25:53.461: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:25:53.461: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:25:54.107: INFO: Latency metrics for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:25:54.108: INFO: Logging node info for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:25:54.287: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-57.ap-northeast-2.compute.internal 2ea0d3e0-dcd9-4562-ada8-ee5e6b866480 5190 0 2022-12-01 23:17:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-57.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-6520":"csi-mock-csi-mock-volumes-6520","ebs.csi.aws.com":"i-02200981456309a72"} io.cilium.network.ipv4-cilium-host:172.20.51.99 io.cilium.network.ipv4-health-ip:172.20.36.120 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:44 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:21:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:22:06 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-01 23:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02200981456309a72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:44 +0000 UTC,LastTransitionTime:2022-12-01 23:17:44 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:14 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:14 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:14 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:22:14 +0000 UTC,LastTransitionTime:2022-12-01 23:17:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.57,},NodeAddress{Type:ExternalIP,Address:3.36.130.97,},NodeAddress{Type:Hostname,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-36-130-97.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bf2fd9fd22cd1aac5da0aa67f7a08,SystemUUID:ec2bf2fd-9fd2-2cd1-aac5-da0aa67f7a08,BootID:c4c37f48-1b62-417d-82bf-ea460350f712,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-6520^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-6520^4,DevicePath:,},},Config:nil,},} Dec 1 23:25:54.287: INFO: Logging kubelet events for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:25:54.467: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:25:54.828: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-ddtfs started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:25:54.828: INFO: cilium-nwg58 started at 2022-12-01 23:17:15 +0000 UTC (1+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:25:54.828: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:25:54.828: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-8l4bh started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:54.828: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-p8zf2 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:54.828: INFO: csi-mockplugin-attacher-0 started at 2022-12-01 23:21:21 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:25:54.828: INFO: pod-509be3d7-a5cc-4c3b-9545-4672553ea64b started at 2022-12-01 23:21:42 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container write-pod ready: false, restart count 0 Dec 1 23:25:54.828: INFO: coredns-5fcc7b6498-x9kjd started at 2022-12-01 23:17:56 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container coredns ready: true, restart count 0 Dec 1 23:25:54.828: INFO: pvc-volume-tester-fgcdz started at 2022-12-01 23:22:05 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container volume-tester ready: false, restart count 0 Dec 1 23:25:54.828: INFO: netserver-2 started at 2022-12-01 23:22:25 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container webserver ready: false, restart count 0 Dec 1 23:25:54.828: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-nhwvz started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:54.828: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-fmm4p started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:54.828: INFO: hostexec-ip-172-20-55-57.ap-northeast-2.compute.internal-l4k9q started at 2022-12-01 23:21:38 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:25:54.828: INFO: webserver-deployment-847dcfb7fb-ddpw2 started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container httpd ready: false, restart count 0 Dec 1 23:25:54.828: INFO: webserver-deployment-847dcfb7fb-7mvn2 started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container httpd ready: false, restart count 0 Dec 1 23:25:54.828: INFO: csi-mockplugin-0 started at 2022-12-01 23:21:48 +0000 UTC (0+3 container statuses recorded) Dec 1 23:25:54.828: INFO: Container csi-provisioner ready: false, restart count 0 Dec 1 23:25:54.828: INFO: Container driver-registrar ready: false, restart count 0 Dec 1 23:25:54.828: INFO: Container mock ready: false, restart count 0 Dec 1 23:25:54.828: INFO: netserver-2 started at 2022-12-01 23:21:41 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container webserver ready: true, restart count 0 Dec 1 23:25:54.828: INFO: pod-subpath-test-preprovisionedpv-5bnz started at 2022-12-01 23:21:48 +0000 UTC (1+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Init container init-volume-preprovisionedpv-5bnz ready: false, restart count 0 Dec 1 23:25:54.828: INFO: Container test-container-subpath-preprovisionedpv-5bnz ready: false, restart count 0 Dec 1 23:25:54.828: INFO: hostexec-ip-172-20-55-57.ap-northeast-2.compute.internal-mj96t started at 2022-12-01 23:21:37 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:25:54.828: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-8gc5h started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:54.828: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-5z2kc started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:54.828: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-nbwwt started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:25:54.828: INFO: affinity-clusterip-transition-g25k2 started at 2022-12-01 23:21:24 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container affinity-clusterip-transition ready: true, restart count 0 Dec 1 23:25:54.828: INFO: csi-mockplugin-attacher-0 started at 2022-12-01 23:21:48 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container csi-attacher ready: false, restart count 0 Dec 1 23:25:54.828: INFO: netserver-2 started at 2022-12-01 23:22:07 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container webserver ready: false, restart count 0 Dec 1 23:25:54.828: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-wzpt2 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:54.828: INFO: csi-mockplugin-0 started at 2022-12-01 23:21:21 +0000 UTC (0+3 container statuses recorded) Dec 1 23:25:54.828: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:25:54.828: INFO: Container driver-registrar ready: true, restart count 0 Dec 1 23:25:54.828: INFO: Container mock ready: true, restart count 0 Dec 1 23:25:54.828: INFO: ebs-csi-node-5zdr7 started at 2022-12-01 23:17:15 +0000 UTC (0+3 container statuses recorded) Dec 1 23:25:54.828: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:25:54.828: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:25:54.828: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:25:54.828: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-vdxsm started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:54.828: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:25:55.449: INFO: Latency metrics for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:25:55.449: INFO: Logging node info for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:25:55.626: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-60-229.ap-northeast-2.compute.internal 63277852-d67d-402c-b963-ae3a2995d791 5538 0 2022-12-01 23:17:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-60-229.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-60-229.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-9901":"ip-172-20-60-229.ap-northeast-2.compute.internal","ebs.csi.aws.com":"i-0441b4e9d8ac12445"} io.cilium.network.ipv4-cilium-host:172.20.50.202 io.cilium.network.ipv4-health-ip:172.20.42.191 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:45 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:21:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2022-12-01 23:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-0441b4e9d8ac12445,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:45 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:22:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.229,},NodeAddress{Type:ExternalIP,Address:43.201.70.39,},NodeAddress{Type:Hostname,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-43-201-70-39.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec24f7651bea0c9073fca065a65b3820,SystemUUID:ec24f765-1bea-0c90-73fc-a065a65b3820,BootID:5386d7df-0b64-4c56-85ff-3f8a9a84c900,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:25:55.626: INFO: Logging kubelet events for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:25:55.806: INFO: Logging pods the kubelet thinks is on node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:25:55.991: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-gvcsf started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.991: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:55.991: INFO: netserver-3 started at 2022-12-01 23:21:42 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container webserver ready: true, restart count 0 Dec 1 23:25:55.992: INFO: alpine-nnp-false-25c04ee9-5657-407e-bd2d-7677b77176ce started at 2022-12-01 23:21:44 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container alpine-nnp-false-25c04ee9-5657-407e-bd2d-7677b77176ce ready: false, restart count 0 Dec 1 23:25:55.992: INFO: csi-mockplugin-0 started at 2022-12-01 23:21:28 +0000 UTC (0+3 container statuses recorded) Dec 1 23:25:55.992: INFO: Container csi-provisioner ready: false, restart count 0 Dec 1 23:25:55.992: INFO: Container driver-registrar ready: false, restart count 0 Dec 1 23:25:55.992: INFO: Container mock ready: false, restart count 0 Dec 1 23:25:55.992: INFO: host-test-container-pod started at 2022-12-01 23:22:06 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:25:55.992: INFO: cilium-5dnwd started at 2022-12-01 23:17:12 +0000 UTC (1+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:25:55.992: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:25:55.992: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-4p7lw started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:55.992: INFO: netserver-3 started at 2022-12-01 23:22:07 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container webserver ready: false, restart count 0 Dec 1 23:25:55.992: INFO: webserver-deployment-847dcfb7fb-pjghl started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container httpd ready: false, restart count 0 Dec 1 23:25:55.992: INFO: execpod-affinitytn96q started at 2022-12-01 23:22:01 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container agnhost-container ready: false, restart count 0 Dec 1 23:25:55.992: INFO: csi-mockplugin-attacher-0 started at 2022-12-01 23:21:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container csi-attacher ready: false, restart count 0 Dec 1 23:25:55.992: INFO: termination-message-containerdbb83236-c389-44a6-8381-8575f82835cc started at 2022-12-01 23:22:05 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container termination-message-container ready: false, restart count 0 Dec 1 23:25:55.992: INFO: sample-webhook-deployment-5d5bc8ffd5-4frfk started at 2022-12-01 23:22:06 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container sample-webhook ready: true, restart count 0 Dec 1 23:25:55.992: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-g98kz started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:25:55.992: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-7vclf started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:55.992: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-2hlfg started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:55.992: INFO: affinity-clusterip-transition-r8gd2 started at 2022-12-01 23:21:24 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container affinity-clusterip-transition ready: true, restart count 0 Dec 1 23:25:55.992: INFO: csi-hostpathplugin-0 started at 2022-12-01 23:21:57 +0000 UTC (0+7 container statuses recorded) Dec 1 23:25:55.992: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:25:55.992: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:25:55.992: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:25:55.992: INFO: Container csi-snapshotter ready: true, restart count 0 Dec 1 23:25:55.992: INFO: Container hostpath ready: true, restart count 0 Dec 1 23:25:55.992: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:25:55.992: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:25:55.992: INFO: inline-volume-tester-8gmln started at 2022-12-01 23:21:58 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container csi-volume-tester ready: false, restart count 0 Dec 1 23:25:55.992: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-zvfn4 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:55.992: INFO: alpine-nnp-true-e2ab7c9b-593a-4110-af8d-d26b2a9ae321 started at 2022-12-01 23:22:00 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container alpine-nnp-true-e2ab7c9b-593a-4110-af8d-d26b2a9ae321 ready: false, restart count 0 Dec 1 23:25:55.992: INFO: to-be-attached-pod started at 2022-12-01 23:22:22 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container container1 ready: false, restart count 0 Dec 1 23:25:55.992: INFO: netserver-3 started at 2022-12-01 23:22:25 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container webserver ready: false, restart count 0 Dec 1 23:25:55.992: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-szrp5 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:55.992: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-fczc8 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:25:55.992: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-nr4j5 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:55.992: INFO: ebs-csi-node-wqh44 started at 2022-12-01 23:17:12 +0000 UTC (0+3 container statuses recorded) Dec 1 23:25:55.992: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:25:55.992: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:25:55.992: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:25:55.992: INFO: test-container-pod started at 2022-12-01 23:22:05 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container webserver ready: false, restart count 0 Dec 1 23:25:55.992: INFO: webserver-deployment-847dcfb7fb-wkgvk started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container httpd ready: false, restart count 0 Dec 1 23:25:55.992: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-c4nx7 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:25:55.992: INFO: csi-mockplugin-resizer-0 started at 2022-12-01 23:21:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container csi-resizer ready: false, restart count 0 Dec 1 23:25:55.992: INFO: pod-handle-http-request started at 2022-12-01 23:21:56 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container agnhost-container ready: false, restart count 0 Dec 1 23:25:55.992: INFO: webserver-deployment-847dcfb7fb-xgjww started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:25:55.992: INFO: Container httpd ready: false, restart count 0 Dec 1 23:25:56.615: INFO: Latency metrics for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:25:56.615: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volumemode-8642" for this suite.
Filter through log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(block\svolmode\)\]\svolumes\sshould\sstore\sdata$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:159 Dec 1 23:48:09.128: Failed to create injector pod: timed out waiting for the condition /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186from junit_08.xml
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumes /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumes /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Dec 1 23:43:06.591: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename volume �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should store data /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:159 Dec 1 23:43:07.479: INFO: Creating resource for dynamic PV Dec 1 23:43:07.479: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass volume-3057-e2e-sc84btj �[1mSTEP�[0m: creating a claim �[1mSTEP�[0m: starting external-injector �[1mSTEP�[0m: Deleting pod external-injector in namespace volume-3057 Dec 1 23:48:08.947: INFO: Waiting for pod external-injector to disappear Dec 1 23:48:09.128: INFO: Pod external-injector no longer exists Dec 1 23:48:09.128: FAIL: Failed to create injector pod: timed out waiting for the condition Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumesTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186 +0x3ff k8s.io/kubernetes/test/e2e.RunE2ETests(0xc00046b680) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc00046b680) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc00046b680, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 �[1mSTEP�[0m: cleaning the environment after external �[1mSTEP�[0m: Deleting pvc Dec 1 23:48:09.306: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.com698nr" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (block volmode)] volumes /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "volume-3057". �[1mSTEP�[0m: Found 6 events. Dec 1 23:48:09.842: INFO: At 2022-12-01 23:43:07 +0000 UTC - event for ebs.csi.aws.com698nr: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Dec 1 23:48:09.842: INFO: At 2022-12-01 23:43:08 +0000 UTC - event for ebs.csi.aws.com698nr: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Dec 1 23:48:09.842: INFO: At 2022-12-01 23:43:08 +0000 UTC - event for ebs.csi.aws.com698nr: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } Provisioning: External provisioner is provisioning volume for claim "volume-3057/ebs.csi.aws.com698nr" Dec 1 23:48:09.842: INFO: At 2022-12-01 23:43:18 +0000 UTC - event for ebs.csi.aws.com698nr: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "volume-3057-e2e-sc84btj": rpc error: code = DeadlineExceeded desc = context deadline exceeded Dec 1 23:48:09.842: INFO: At 2022-12-01 23:44:13 +0000 UTC - event for ebs.csi.aws.com698nr: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "volume-3057-e2e-sc84btj": rpc error: code = Internal desc = Could not create volume "pvc-4bf54661-7620-4db0-8c9e-2a4962808dd3": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Dec 1 23:48:09.842: INFO: At 2022-12-01 23:48:09 +0000 UTC - event for external-injector: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "external-injector" not found Dec 1 23:48:10.020: INFO: POD NODE PHASE GRACE CONDITIONS Dec 1 23:48:10.020: INFO: Dec 1 23:48:10.199: INFO: Logging node info for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:48:10.376: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-156.ap-northeast-2.compute.internal e92fe5b6-ec1b-421a-992a-1182ff1e3522 32516 0 2022-12-01 23:15:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-156.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03fe1aa2fac33129f"} io.cilium.network.ipv4-cilium-host:172.20.40.211 io.cilium.network.ipv4-health-ip:172.20.46.196 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-12-01 23:15:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-12-01 23:15:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-12-01 23:15:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:16:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:21 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03fe1aa2fac33129f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:21 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:48:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:48:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:48:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:48:02 +0000 UTC,LastTransitionTime:2022-12-01 23:16:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.156,},NodeAddress{Type:ExternalIP,Address:15.165.75.247,},NodeAddress{Type:Hostname,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-165-75-247.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccbd8e0eb399c0115bc276d51c7ab,SystemUUID:ec2ccbd8-e0eb-399c-0115-bc276d51c7ab,BootID:4c45c6e5-af61-46fe-b037-b7cc919a53fc,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.2],SizeBytes:41958567,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.2],SizeBytes:41441104,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.2],SizeBytes:4954598,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:48:10.377: INFO: Logging kubelet events for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:48:10.557: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:48:10.740: INFO: kube-scheduler-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:10.740: INFO: Container kube-scheduler ready: true, restart count 0 Dec 1 23:48:10.740: INFO: kube-apiserver-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+2 container statuses recorded) Dec 1 23:48:10.740: INFO: Container healthcheck ready: true, restart count 0 Dec 1 23:48:10.740: INFO: Container kube-apiserver ready: true, restart count 1 Dec 1 23:48:10.740: INFO: ebs-csi-node-m7cgg started at 2022-12-01 23:15:46 +0000 UTC (0+3 container statuses recorded) Dec 1 23:48:10.740: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:48:10.740: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:48:10.740: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:48:10.740: INFO: dns-controller-5fb778d9d5-qrfkv started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:10.740: INFO: Container dns-controller ready: true, restart count 0 Dec 1 23:48:10.740: INFO: cilium-operator-587dd66f87-pgs67 started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:10.740: INFO: Container cilium-operator ready: true, restart count 0 Dec 1 23:48:10.740: INFO: kube-controller-manager-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:10.740: INFO: Container kube-controller-manager ready: true, restart count 2 Dec 1 23:48:10.740: INFO: etcd-manager-events-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:10.740: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:48:10.740: INFO: etcd-manager-main-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:10.740: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:48:10.740: INFO: kops-controller-g4pqg started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:10.740: INFO: Container kops-controller ready: true, restart count 0 Dec 1 23:48:10.740: INFO: cilium-pb6vg started at 2022-12-01 23:15:46 +0000 UTC (1+1 container statuses recorded) Dec 1 23:48:10.740: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:48:10.740: INFO: Container cilium-agent ready: true, restart count 1 Dec 1 23:48:10.740: INFO: ebs-csi-controller-764b64d679-5qsrv started at 2022-12-01 23:15:46 +0000 UTC (0+5 container statuses recorded) Dec 1 23:48:10.740: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:48:10.740: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:48:10.740: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:48:10.740: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:48:10.740: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:48:11.346: INFO: Latency metrics for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:48:11.346: INFO: Logging node info for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:48:11.523: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-46-84.ap-northeast-2.compute.internal e408f2b8-c522-4e88-973a-78a2b7a2298f 28204 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-46-84.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-3465 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-46-84.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03a01a80f538d5d6b"} io.cilium.network.ipv4-cilium-host:172.20.38.72 io.cilium.network.ipv4-health-ip:172.20.48.223 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:40 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:40:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kubelet Update v1 2022-12-01 23:46:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03a01a80f538d5d6b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:40 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:46:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.46.84,},NodeAddress{Type:ExternalIP,Address:54.180.162.145,},NodeAddress{Type:Hostname,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-162-145.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec285e4d0d3397ed7e702d489e42654b,SystemUUID:ec285e4d-0d33-97ed-7e70-2d489e42654b,BootID:d6d1d15f-8121-4e40-b71c-e5b51419dc1b,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:48:11.524: INFO: Logging kubelet events for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:48:11.704: INFO: Logging pods the kubelet thinks is on node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:48:11.884: INFO: ebs-csi-node-lfxzl started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:48:11.884: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:48:11.884: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:48:11.884: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:48:11.884: INFO: cilium-7h664 started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:48:11.884: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:48:11.884: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:48:11.884: INFO: startup-714ba7b9-cfb1-40a5-ad4c-f7bfba82598d started at 2022-12-01 23:44:50 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:11.884: INFO: Container busybox ready: false, restart count 0 Dec 1 23:48:11.884: INFO: service-proxy-toggled-226b5 started at 2022-12-01 23:47:33 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:11.884: INFO: Container service-proxy-toggled ready: true, restart count 0 Dec 1 23:48:11.884: INFO: service-proxy-disabled-r4947 started at 2022-12-01 23:47:27 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:11.884: INFO: Container service-proxy-disabled ready: true, restart count 0 Dec 1 23:48:12.493: INFO: Latency metrics for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:48:12.493: INFO: Logging node info for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:48:12.671: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-54-150.ap-northeast-2.compute.internal 2a8a1d71-d00b-4259-82bd-841f96508593 32646 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-54-150.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-54-150.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02d689db76221fe72"} io.cilium.network.ipv4-cilium-host:172.20.43.0 io.cilium.network.ipv4-health-ip:172.20.58.253 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-12-01 23:17:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:38 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:41:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02d689db76221fe72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:38 +0000 UTC,LastTransitionTime:2022-12-01 23:17:38 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:48:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:48:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:48:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:48:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:30 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.150,},NodeAddress{Type:ExternalIP,Address:54.180.133.96,},NodeAddress{Type:Hostname,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-133-96.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec20870a7ff0c6740650f49249abc430,SystemUUID:ec20870a-7ff0-c674-0650-f49249abc430,BootID:803a40bf-5805-4ae9-b933-fde612d75f1d,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 k8s.gcr.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:e7fddbaac4c3451da2365ab90bad149d32f11409738034e41e0f460927f7c276 k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.4],SizeBytes:24757245,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:48:12.672: INFO: Logging kubelet events for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:48:12.852: INFO: Logging pods the kubelet thinks is on node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:48:13.035: INFO: service-proxy-disabled-26m9c started at 2022-12-01 23:47:27 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:13.035: INFO: Container service-proxy-disabled ready: true, restart count 0 Dec 1 23:48:13.035: INFO: service-proxy-toggled-mcm5d started at 2022-12-01 23:47:33 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:13.035: INFO: Container service-proxy-toggled ready: true, restart count 0 Dec 1 23:48:13.035: INFO: coredns-autoscaler-6658b4bf85-92nhw started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:13.035: INFO: Container autoscaler ready: true, restart count 0 Dec 1 23:48:13.035: INFO: implicit-nonroot-uid started at 2022-12-01 23:48:00 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:13.035: INFO: Container implicit-nonroot-uid ready: false, restart count 0 Dec 1 23:48:13.035: INFO: ss2-2 started at 2022-12-01 23:47:51 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:13.035: INFO: Container webserver ready: true, restart count 0 Dec 1 23:48:13.035: INFO: replace-27832308-q8qcb started at 2022-12-01 23:48:00 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:13.035: INFO: Container c ready: true, restart count 0 Dec 1 23:48:13.035: INFO: ebs-csi-node-n5tcr started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:48:13.035: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:48:13.035: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:48:13.035: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:48:13.035: INFO: coredns-5fcc7b6498-vcsml started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:13.035: INFO: Container coredns ready: true, restart count 0 Dec 1 23:48:13.035: INFO: verify-service-down-host-exec-pod started at 2022-12-01 23:48:10 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:13.035: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:48:13.035: INFO: cilium-2bxvd started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:48:13.035: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:48:13.035: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:48:13.647: INFO: Latency metrics for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:48:13.647: INFO: Logging node info for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:48:13.825: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-57.ap-northeast-2.compute.internal 2ea0d3e0-dcd9-4562-ada8-ee5e6b866480 29483 0 2022-12-01 23:17:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-57.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-55-57.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02200981456309a72"} io.cilium.network.ipv4-cilium-host:172.20.51.99 io.cilium.network.ipv4-health-ip:172.20.36.120 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:44 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02200981456309a72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:44 +0000 UTC,LastTransitionTime:2022-12-01 23:17:44 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.57,},NodeAddress{Type:ExternalIP,Address:3.36.130.97,},NodeAddress{Type:Hostname,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-36-130-97.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bf2fd9fd22cd1aac5da0aa67f7a08,SystemUUID:ec2bf2fd-9fd2-2cd1-aac5-da0aa67f7a08,BootID:c4c37f48-1b62-417d-82bf-ea460350f712,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-936^47b47419-71d1-11ed-be23-02bdc8e24a07],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:48:13.826: INFO: Logging kubelet events for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:48:14.007: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:48:14.188: INFO: cilium-nwg58 started at 2022-12-01 23:17:15 +0000 UTC (1+1 container statuses recorded) Dec 1 23:48:14.189: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:48:14.189: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:48:14.189: INFO: coredns-5fcc7b6498-x9kjd started at 2022-12-01 23:17:56 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:14.189: INFO: Container coredns ready: true, restart count 0 Dec 1 23:48:14.189: INFO: hostexec-ip-172-20-55-57.ap-northeast-2.compute.internal-zh99m started at 2022-12-01 23:47:54 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:14.189: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:48:14.189: INFO: hostexec-ip-172-20-55-57.ap-northeast-2.compute.internal-rprnk started at 2022-12-01 23:47:33 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:14.189: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:48:14.189: INFO: local-injector started at 2022-12-01 23:48:03 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:14.189: INFO: Container local-injector ready: true, restart count 0 Dec 1 23:48:14.189: INFO: ss2-0 started at 2022-12-01 23:47:48 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:14.189: INFO: Container webserver ready: true, restart count 0 Dec 1 23:48:14.189: INFO: ebs-csi-node-5zdr7 started at 2022-12-01 23:17:15 +0000 UTC (0+3 container statuses recorded) Dec 1 23:48:14.189: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:48:14.189: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:48:14.189: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:48:14.189: INFO: busybox-ed5fcaf8-791e-42f4-9c29-b9c8d6e1d424 started at 2022-12-01 23:47:38 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:14.189: INFO: Container busybox ready: false, restart count 0 Dec 1 23:48:14.189: INFO: service-proxy-toggled-vznnc started at 2022-12-01 23:47:33 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:14.189: INFO: Container service-proxy-toggled ready: true, restart count 0 Dec 1 23:48:14.189: INFO: pod-subpath-test-preprovisionedpv-4w72 started at 2022-12-01 23:47:48 +0000 UTC (1+1 container statuses recorded) Dec 1 23:48:14.189: INFO: Init container init-volume-preprovisionedpv-4w72 ready: true, restart count 0 Dec 1 23:48:14.189: INFO: Container test-container-subpath-preprovisionedpv-4w72 ready: false, restart count 0 Dec 1 23:48:14.833: INFO: Latency metrics for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:48:14.833: INFO: Logging node info for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:48:15.011: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-60-229.ap-northeast-2.compute.internal 63277852-d67d-402c-b963-ae3a2995d791 29243 0 2022-12-01 23:17:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-60-229.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-60-229.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0441b4e9d8ac12445"} io.cilium.network.ipv4-cilium-host:172.20.50.202 io.cilium.network.ipv4-health-ip:172.20.42.191 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:45 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:41:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-0441b4e9d8ac12445,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:45 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:46:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.229,},NodeAddress{Type:ExternalIP,Address:43.201.70.39,},NodeAddress{Type:Hostname,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-43-201-70-39.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec24f7651bea0c9073fca065a65b3820,SystemUUID:ec24f765-1bea-0c90-73fc-a065a65b3820,BootID:5386d7df-0b64-4c56-85ff-3f8a9a84c900,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:48:15.011: INFO: Logging kubelet events for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:48:15.192: INFO: Logging pods the kubelet thinks is on node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:48:15.374: INFO: ss2-1 started at 2022-12-01 23:47:50 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:15.375: INFO: Container webserver ready: true, restart count 0 Dec 1 23:48:15.375: INFO: ebs-csi-node-wqh44 started at 2022-12-01 23:17:12 +0000 UTC (0+3 container statuses recorded) Dec 1 23:48:15.375: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:48:15.375: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:48:15.375: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:48:15.375: INFO: cilium-5dnwd started at 2022-12-01 23:17:12 +0000 UTC (1+1 container statuses recorded) Dec 1 23:48:15.375: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:48:15.375: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:48:15.375: INFO: hostexec-ip-172-20-60-229.ap-northeast-2.compute.internal-b9km9 started at 2022-12-01 23:47:59 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:15.375: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:48:15.375: INFO: service-proxy-disabled-9ltgd started at 2022-12-01 23:47:27 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:15.375: INFO: Container service-proxy-disabled ready: true, restart count 0 Dec 1 23:48:15.375: INFO: hostexec-ip-172-20-60-229.ap-northeast-2.compute.internal-cgl82 started at 2022-12-01 23:48:04 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:15.375: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:48:15.976: INFO: Latency metrics for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:48:15.976: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volume-3057" for this suite.
Filter through log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\(allowExpansion\)\]\svolume\-expand\sVerify\sif\soffline\sPVC\sexpansion\sworks$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:174 Dec 1 23:51:08.204: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc0043b6fe0>: { s: "pod \"pod-408ea52d-06d0-404f-805f-eddba33b38a0\" is not Running: timed out waiting for the condition", } pod "pod-408ea52d-06d0-404f-805f-eddba33b38a0" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:192from junit_25.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Dec 1 23:46:06.062: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename volume-expand �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] Verify if offline PVC expansion works /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:174 Dec 1 23:46:06.948: INFO: Creating resource for dynamic PV Dec 1 23:46:06.948: INFO: Using claimSize:1Gi, test suite supported size:{ 1Gi}, driver(ebs.csi.aws.com) supported size:{ 1Gi} �[1mSTEP�[0m: creating a StorageClass volume-expand-7548-e2e-sc8d82h �[1mSTEP�[0m: creating a claim Dec 1 23:46:07.127: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating a pod with dynamically provisioned volume Dec 1 23:51:08.204: FAIL: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc0043b6fe0>: { s: "pod \"pod-408ea52d-06d0-404f-805f-eddba33b38a0\" is not Running: timed out waiting for the condition", } pod "pod-408ea52d-06d0-404f-805f-eddba33b38a0" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func4() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:192 +0x34d k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000afcd80) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000afcd80) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000afcd80, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 Dec 1 23:51:08.204: INFO: Deleting pod "pod-408ea52d-06d0-404f-805f-eddba33b38a0" in namespace "volume-expand-7548" Dec 1 23:51:08.387: INFO: Wait up to 5m0s for pod "pod-408ea52d-06d0-404f-805f-eddba33b38a0" to be fully deleted �[1mSTEP�[0m: Deleting pod Dec 1 23:51:08.564: INFO: Deleting pod "pod-408ea52d-06d0-404f-805f-eddba33b38a0" in namespace "volume-expand-7548" �[1mSTEP�[0m: Deleting pvc Dec 1 23:51:08.919: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.com2ll6b" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "volume-expand-7548". �[1mSTEP�[0m: Found 5 events. Dec 1 23:51:09.455: INFO: At 2022-12-01 23:46:07 +0000 UTC - event for ebs.csi.aws.com2ll6b: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Dec 1 23:51:09.455: INFO: At 2022-12-01 23:46:07 +0000 UTC - event for ebs.csi.aws.com2ll6b: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Dec 1 23:51:09.455: INFO: At 2022-12-01 23:46:07 +0000 UTC - event for ebs.csi.aws.com2ll6b: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-7548/ebs.csi.aws.com2ll6b" Dec 1 23:51:09.455: INFO: At 2022-12-01 23:46:17 +0000 UTC - event for ebs.csi.aws.com2ll6b: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-7548-e2e-sc8d82h": rpc error: code = DeadlineExceeded desc = context deadline exceeded Dec 1 23:51:09.455: INFO: At 2022-12-01 23:51:08 +0000 UTC - event for pod-408ea52d-06d0-404f-805f-eddba33b38a0: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "pod-408ea52d-06d0-404f-805f-eddba33b38a0" not found Dec 1 23:51:09.632: INFO: POD NODE PHASE GRACE CONDITIONS Dec 1 23:51:09.632: INFO: Dec 1 23:51:09.811: INFO: Logging node info for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:51:09.992: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-156.ap-northeast-2.compute.internal e92fe5b6-ec1b-421a-992a-1182ff1e3522 32516 0 2022-12-01 23:15:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-156.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03fe1aa2fac33129f"} io.cilium.network.ipv4-cilium-host:172.20.40.211 io.cilium.network.ipv4-health-ip:172.20.46.196 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-12-01 23:15:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-12-01 23:15:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-12-01 23:15:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:16:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:21 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03fe1aa2fac33129f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:21 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:48:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:48:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:48:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:48:02 +0000 UTC,LastTransitionTime:2022-12-01 23:16:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.156,},NodeAddress{Type:ExternalIP,Address:15.165.75.247,},NodeAddress{Type:Hostname,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-165-75-247.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccbd8e0eb399c0115bc276d51c7ab,SystemUUID:ec2ccbd8-e0eb-399c-0115-bc276d51c7ab,BootID:4c45c6e5-af61-46fe-b037-b7cc919a53fc,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.2],SizeBytes:41958567,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.2],SizeBytes:41441104,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.2],SizeBytes:4954598,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:51:09.994: INFO: Logging kubelet events for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:51:10.174: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:51:10.359: INFO: kube-scheduler-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:10.359: INFO: Container kube-scheduler ready: true, restart count 0 Dec 1 23:51:10.359: INFO: kube-apiserver-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+2 container statuses recorded) Dec 1 23:51:10.359: INFO: Container healthcheck ready: true, restart count 0 Dec 1 23:51:10.359: INFO: Container kube-apiserver ready: true, restart count 1 Dec 1 23:51:10.359: INFO: ebs-csi-node-m7cgg started at 2022-12-01 23:15:46 +0000 UTC (0+3 container statuses recorded) Dec 1 23:51:10.359: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:51:10.359: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:51:10.359: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:51:10.359: INFO: dns-controller-5fb778d9d5-qrfkv started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:10.359: INFO: Container dns-controller ready: true, restart count 0 Dec 1 23:51:10.359: INFO: cilium-operator-587dd66f87-pgs67 started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:10.359: INFO: Container cilium-operator ready: true, restart count 0 Dec 1 23:51:10.359: INFO: kube-controller-manager-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:10.359: INFO: Container kube-controller-manager ready: true, restart count 2 Dec 1 23:51:10.359: INFO: etcd-manager-events-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:10.359: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:51:10.359: INFO: etcd-manager-main-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:10.359: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:51:10.359: INFO: kops-controller-g4pqg started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:10.359: INFO: Container kops-controller ready: true, restart count 0 Dec 1 23:51:10.359: INFO: cilium-pb6vg started at 2022-12-01 23:15:46 +0000 UTC (1+1 container statuses recorded) Dec 1 23:51:10.359: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:51:10.359: INFO: Container cilium-agent ready: true, restart count 1 Dec 1 23:51:10.359: INFO: ebs-csi-controller-764b64d679-5qsrv started at 2022-12-01 23:15:46 +0000 UTC (0+5 container statuses recorded) Dec 1 23:51:10.359: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:51:10.359: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:51:10.359: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:51:10.359: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:51:10.359: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:51:10.953: INFO: Latency metrics for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:51:10.953: INFO: Logging node info for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:51:11.131: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-46-84.ap-northeast-2.compute.internal e408f2b8-c522-4e88-973a-78a2b7a2298f 34372 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-46-84.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-3465 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-46-84.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-712":"csi-mock-csi-mock-volumes-712","ebs.csi.aws.com":"i-03a01a80f538d5d6b"} io.cilium.network.ipv4-cilium-host:172.20.38.72 io.cilium.network.ipv4-health-ip:172.20.48.223 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:40 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:40:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:48:54 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-01 23:49:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03a01a80f538d5d6b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:40 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:49:01 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:49:01 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:49:01 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:49:01 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.46.84,},NodeAddress{Type:ExternalIP,Address:54.180.162.145,},NodeAddress{Type:Hostname,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-162-145.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec285e4d0d3397ed7e702d489e42654b,SystemUUID:ec285e4d-0d33-97ed-7e70-2d489e42654b,BootID:d6d1d15f-8121-4e40-b71c-e5b51419dc1b,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-712^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-712^4,DevicePath:,},},Config:nil,},} Dec 1 23:51:11.132: INFO: Logging kubelet events for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:51:11.312: INFO: Logging pods the kubelet thinks is on node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:51:11.493: INFO: ebs-csi-node-lfxzl started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:51:11.493: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:51:11.493: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:51:11.493: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:51:11.493: INFO: csi-mockplugin-attacher-0 started at 2022-12-01 23:48:47 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:11.493: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:51:11.493: INFO: cilium-7h664 started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:51:11.493: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:51:11.493: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:51:11.493: INFO: csi-mockplugin-0 started at 2022-12-01 23:48:47 +0000 UTC (0+3 container statuses recorded) Dec 1 23:51:11.493: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:51:11.493: INFO: Container driver-registrar ready: true, restart count 0 Dec 1 23:51:11.493: INFO: Container mock ready: true, restart count 0 Dec 1 23:51:11.493: INFO: affinity-nodeport-timeout-h2px9 started at 2022-12-01 23:49:43 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:11.493: INFO: Container affinity-nodeport-timeout ready: true, restart count 0 Dec 1 23:51:12.115: INFO: Latency metrics for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:51:12.115: INFO: Logging node info for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:51:12.294: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-54-150.ap-northeast-2.compute.internal 2a8a1d71-d00b-4259-82bd-841f96508593 36203 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-54-150.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-54-150.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-1064":"csi-mock-csi-mock-volumes-1064","ebs.csi.aws.com":"i-02d689db76221fe72"} io.cilium.network.ipv4-cilium-host:172.20.43.0 io.cilium.network.ipv4-health-ip:172.20.58.253 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-12-01 23:17:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:38 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:49:41 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02d689db76221fe72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:38 +0000 UTC,LastTransitionTime:2022-12-01 23:17:38 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:49:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:49:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:49:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:49:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:30 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.150,},NodeAddress{Type:ExternalIP,Address:54.180.133.96,},NodeAddress{Type:Hostname,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-133-96.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec20870a7ff0c6740650f49249abc430,SystemUUID:ec20870a-7ff0-c674-0650-f49249abc430,BootID:803a40bf-5805-4ae9-b933-fde612d75f1d,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 k8s.gcr.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:e7fddbaac4c3451da2365ab90bad149d32f11409738034e41e0f460927f7c276 k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.4],SizeBytes:24757245,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:51:12.295: INFO: Logging kubelet events for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:51:12.475: INFO: Logging pods the kubelet thinks is on node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:51:12.656: INFO: coredns-autoscaler-6658b4bf85-92nhw started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:12.656: INFO: Container autoscaler ready: true, restart count 0 Dec 1 23:51:12.656: INFO: hostexec-ip-172-20-54-150.ap-northeast-2.compute.internal-nnkwf started at 2022-12-01 23:51:03 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:12.656: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:51:12.656: INFO: csi-mockplugin-0 started at 2022-12-01 23:50:03 +0000 UTC (0+3 container statuses recorded) Dec 1 23:51:12.656: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:51:12.656: INFO: Container driver-registrar ready: true, restart count 0 Dec 1 23:51:12.656: INFO: Container mock ready: true, restart count 0 Dec 1 23:51:12.656: INFO: csi-mockplugin-attacher-0 started at 2022-12-01 23:50:03 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:12.656: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:51:12.656: INFO: pvc-volume-tester-f8mcp started at 2022-12-01 23:50:09 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:12.656: INFO: Container volume-tester ready: true, restart count 0 Dec 1 23:51:12.656: INFO: ebs-csi-node-n5tcr started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:51:12.656: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:51:12.656: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:51:12.656: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:51:12.656: INFO: coredns-5fcc7b6498-vcsml started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:12.656: INFO: Container coredns ready: true, restart count 0 Dec 1 23:51:12.656: INFO: cilium-2bxvd started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:51:12.656: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:51:12.656: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:51:13.265: INFO: Latency metrics for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:51:13.266: INFO: Logging node info for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:51:13.443: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-57.ap-northeast-2.compute.internal 2ea0d3e0-dcd9-4562-ada8-ee5e6b866480 35283 0 2022-12-01 23:17:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-57.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-55-57.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02200981456309a72"} io.cilium.network.ipv4-cilium-host:172.20.51.99 io.cilium.network.ipv4-health-ip:172.20.36.120 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:44 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02200981456309a72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:44 +0000 UTC,LastTransitionTime:2022-12-01 23:17:44 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.57,},NodeAddress{Type:ExternalIP,Address:3.36.130.97,},NodeAddress{Type:Hostname,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-36-130-97.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bf2fd9fd22cd1aac5da0aa67f7a08,SystemUUID:ec2bf2fd-9fd2-2cd1-aac5-da0aa67f7a08,BootID:c4c37f48-1b62-417d-82bf-ea460350f712,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-936^47b47419-71d1-11ed-be23-02bdc8e24a07],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:51:13.444: INFO: Logging kubelet events for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:51:13.624: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:51:13.808: INFO: execpod-affinityzwr72 started at 2022-12-01 23:49:47 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:13.808: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:51:13.808: INFO: httpd started at 2022-12-01 23:51:02 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:13.808: INFO: Container httpd ready: true, restart count 0 Dec 1 23:51:13.808: INFO: ebs-csi-node-5zdr7 started at 2022-12-01 23:17:15 +0000 UTC (0+3 container statuses recorded) Dec 1 23:51:13.808: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:51:13.808: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:51:13.808: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:51:13.808: INFO: affinity-nodeport-timeout-4p5ws started at 2022-12-01 23:49:43 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:13.808: INFO: Container affinity-nodeport-timeout ready: true, restart count 0 Dec 1 23:51:13.808: INFO: inline-volume-wsn92 started at 2022-12-01 23:50:12 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:13.808: INFO: Container volume-tester ready: false, restart count 0 Dec 1 23:51:13.808: INFO: alpine-nnp-nil-6742885a-d4bb-4c33-8383-52e8cc7a3670 started at 2022-12-01 23:51:05 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:13.808: INFO: Container alpine-nnp-nil-6742885a-d4bb-4c33-8383-52e8cc7a3670 ready: false, restart count 0 Dec 1 23:51:13.808: INFO: hostexec-ip-172-20-55-57.ap-northeast-2.compute.internal-dw4bs started at 2022-12-01 23:50:56 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:13.808: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:51:13.808: INFO: csi-hostpathplugin-0 started at 2022-12-01 23:51:12 +0000 UTC (0+7 container statuses recorded) Dec 1 23:51:13.808: INFO: Container csi-attacher ready: false, restart count 0 Dec 1 23:51:13.808: INFO: Container csi-provisioner ready: false, restart count 0 Dec 1 23:51:13.808: INFO: Container csi-resizer ready: false, restart count 0 Dec 1 23:51:13.808: INFO: Container csi-snapshotter ready: false, restart count 0 Dec 1 23:51:13.808: INFO: Container hostpath ready: false, restart count 0 Dec 1 23:51:13.808: INFO: Container liveness-probe ready: false, restart count 0 Dec 1 23:51:13.808: INFO: Container node-driver-registrar ready: false, restart count 0 Dec 1 23:51:13.808: INFO: cilium-nwg58 started at 2022-12-01 23:17:15 +0000 UTC (1+1 container statuses recorded) Dec 1 23:51:13.808: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:51:13.808: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:51:13.808: INFO: coredns-5fcc7b6498-x9kjd started at 2022-12-01 23:17:56 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:13.808: INFO: Container coredns ready: true, restart count 0 Dec 1 23:51:14.507: INFO: Latency metrics for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:51:14.507: INFO: Logging node info for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:51:14.685: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-60-229.ap-northeast-2.compute.internal 63277852-d67d-402c-b963-ae3a2995d791 37399 0 2022-12-01 23:17:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-60-229.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-60-229.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-8363":"ip-172-20-60-229.ap-northeast-2.compute.internal","ebs.csi.aws.com":"i-0441b4e9d8ac12445"} io.cilium.network.ipv4-cilium-host:172.20.50.202 io.cilium.network.ipv4-health-ip:172.20.42.191 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:45 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-12-01 23:51:00 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-01 23:51:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-0441b4e9d8ac12445,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:45 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:51:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:51:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:51:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:51:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.229,},NodeAddress{Type:ExternalIP,Address:43.201.70.39,},NodeAddress{Type:Hostname,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-43-201-70-39.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec24f7651bea0c9073fca065a65b3820,SystemUUID:ec24f765-1bea-0c90-73fc-a065a65b3820,BootID:5386d7df-0b64-4c56-85ff-3f8a9a84c900,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-8363^00f92ba1-71d3-11ed-9775-36c4965dec94],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-8363^00f92ba1-71d3-11ed-9775-36c4965dec94,DevicePath:,},},Config:nil,},} Dec 1 23:51:14.685: INFO: Logging kubelet events for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:51:14.866: INFO: Logging pods the kubelet thinks is on node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:51:15.047: INFO: pod-subpath-test-dynamicpv-qf5v started at 2022-12-01 23:51:11 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:15.047: INFO: Container test-container-subpath-dynamicpv-qf5v ready: false, restart count 0 Dec 1 23:51:15.047: INFO: csi-hostpathplugin-0 started at 2022-12-01 23:50:56 +0000 UTC (0+7 container statuses recorded) Dec 1 23:51:15.047: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:51:15.047: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:51:15.047: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:51:15.047: INFO: Container csi-snapshotter ready: true, restart count 0 Dec 1 23:51:15.047: INFO: Container hostpath ready: true, restart count 0 Dec 1 23:51:15.047: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:51:15.047: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:51:15.047: INFO: affinity-nodeport-timeout-bvbhm started at 2022-12-01 23:49:43 +0000 UTC (0+1 container statuses recorded) Dec 1 23:51:15.047: INFO: Container affinity-nodeport-timeout ready: true, restart count 0 Dec 1 23:51:15.047: INFO: ebs-csi-node-wqh44 started at 2022-12-01 23:17:12 +0000 UTC (0+3 container statuses recorded) Dec 1 23:51:15.047: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:51:15.047: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:51:15.047: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:51:15.047: INFO: cilium-5dnwd started at 2022-12-01 23:17:12 +0000 UTC (1+1 container statuses recorded) Dec 1 23:51:15.047: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:51:15.047: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:51:15.671: INFO: Latency metrics for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:51:15.671: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volume-expand-7548" for this suite.
Find pod-408ea52d-06d0-404f-805f-eddba33b38a0 mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\(allowExpansion\)\]\svolume\-expand\sshould\sresize\svolume\swhen\sPVC\sis\sedited\swhile\spod\sis\susing\sit$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:246 Dec 1 23:43:48.087: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc0036d9710>: { s: "pod \"pod-d990c8d7-602a-45c5-8490-2510698d5907\" is not Running: timed out waiting for the condition", } pod "pod-d990c8d7-602a-45c5-8490-2510698d5907" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:268from junit_14.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Dec 1 23:38:45.963: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename volume-expand �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should resize volume when PVC is edited while pod is using it /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:246 Dec 1 23:38:46.847: INFO: Creating resource for dynamic PV Dec 1 23:38:46.847: INFO: Using claimSize:1Gi, test suite supported size:{ 1Gi}, driver(ebs.csi.aws.com) supported size:{ 1Gi} �[1mSTEP�[0m: creating a StorageClass volume-expand-9861-e2e-scrsg4h �[1mSTEP�[0m: creating a claim Dec 1 23:38:47.024: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating a pod with dynamically provisioned volume Dec 1 23:43:48.087: FAIL: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc0036d9710>: { s: "pod \"pod-d990c8d7-602a-45c5-8490-2510698d5907\" is not Running: timed out waiting for the condition", } pod "pod-d990c8d7-602a-45c5-8490-2510698d5907" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func5() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:268 +0x3d0 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc0002a6780) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc0002a6780) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc0002a6780, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 Dec 1 23:43:48.088: INFO: Deleting pod "pod-d990c8d7-602a-45c5-8490-2510698d5907" in namespace "volume-expand-9861" Dec 1 23:43:48.277: INFO: Wait up to 5m0s for pod "pod-d990c8d7-602a-45c5-8490-2510698d5907" to be fully deleted �[1mSTEP�[0m: Deleting pod Dec 1 23:43:48.454: INFO: Deleting pod "pod-d990c8d7-602a-45c5-8490-2510698d5907" in namespace "volume-expand-9861" �[1mSTEP�[0m: Deleting pvc Dec 1 23:43:48.807: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.coml24xs" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "volume-expand-9861". �[1mSTEP�[0m: Found 6 events. Dec 1 23:43:49.342: INFO: At 2022-12-01 23:38:47 +0000 UTC - event for ebs.csi.aws.coml24xs: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Dec 1 23:43:49.342: INFO: At 2022-12-01 23:38:47 +0000 UTC - event for ebs.csi.aws.coml24xs: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Dec 1 23:43:49.342: INFO: At 2022-12-01 23:38:47 +0000 UTC - event for ebs.csi.aws.coml24xs: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-9861/ebs.csi.aws.coml24xs" Dec 1 23:43:49.342: INFO: At 2022-12-01 23:38:57 +0000 UTC - event for ebs.csi.aws.coml24xs: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-9861-e2e-scrsg4h": rpc error: code = DeadlineExceeded desc = context deadline exceeded Dec 1 23:43:49.343: INFO: At 2022-12-01 23:39:20 +0000 UTC - event for ebs.csi.aws.coml24xs: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-9861-e2e-scrsg4h": rpc error: code = Internal desc = Could not create volume "pvc-c4719719-f4ff-49fa-a846-f02d70e55d9c": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Dec 1 23:43:49.343: INFO: At 2022-12-01 23:43:48 +0000 UTC - event for pod-d990c8d7-602a-45c5-8490-2510698d5907: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "pod-d990c8d7-602a-45c5-8490-2510698d5907" not found Dec 1 23:43:49.519: INFO: POD NODE PHASE GRACE CONDITIONS Dec 1 23:43:49.519: INFO: Dec 1 23:43:49.697: INFO: Logging node info for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:43:49.874: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-156.ap-northeast-2.compute.internal e92fe5b6-ec1b-421a-992a-1182ff1e3522 22013 0 2022-12-01 23:15:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-156.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03fe1aa2fac33129f"} io.cilium.network.ipv4-cilium-host:172.20.40.211 io.cilium.network.ipv4-health-ip:172.20.46.196 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-12-01 23:15:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-12-01 23:15:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-12-01 23:15:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:16:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:21 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03fe1aa2fac33129f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:21 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:43:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:43:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:43:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:43:02 +0000 UTC,LastTransitionTime:2022-12-01 23:16:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.156,},NodeAddress{Type:ExternalIP,Address:15.165.75.247,},NodeAddress{Type:Hostname,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-165-75-247.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccbd8e0eb399c0115bc276d51c7ab,SystemUUID:ec2ccbd8-e0eb-399c-0115-bc276d51c7ab,BootID:4c45c6e5-af61-46fe-b037-b7cc919a53fc,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.2],SizeBytes:41958567,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.2],SizeBytes:41441104,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.2],SizeBytes:4954598,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:43:49.875: INFO: Logging kubelet events for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:43:50.056: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:43:50.237: INFO: ebs-csi-node-m7cgg started at 2022-12-01 23:15:46 +0000 UTC (0+3 container statuses recorded) Dec 1 23:43:50.237: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:43:50.237: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:43:50.237: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:43:50.237: INFO: dns-controller-5fb778d9d5-qrfkv started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:50.237: INFO: Container dns-controller ready: true, restart count 0 Dec 1 23:43:50.237: INFO: cilium-operator-587dd66f87-pgs67 started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:50.237: INFO: Container cilium-operator ready: true, restart count 0 Dec 1 23:43:50.237: INFO: kube-scheduler-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:50.237: INFO: Container kube-scheduler ready: true, restart count 0 Dec 1 23:43:50.237: INFO: kube-apiserver-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+2 container statuses recorded) Dec 1 23:43:50.237: INFO: Container healthcheck ready: true, restart count 0 Dec 1 23:43:50.237: INFO: Container kube-apiserver ready: true, restart count 1 Dec 1 23:43:50.237: INFO: etcd-manager-main-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:50.237: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:43:50.237: INFO: kops-controller-g4pqg started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:50.237: INFO: Container kops-controller ready: true, restart count 0 Dec 1 23:43:50.237: INFO: cilium-pb6vg started at 2022-12-01 23:15:46 +0000 UTC (1+1 container statuses recorded) Dec 1 23:43:50.238: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:43:50.238: INFO: Container cilium-agent ready: true, restart count 1 Dec 1 23:43:50.238: INFO: ebs-csi-controller-764b64d679-5qsrv started at 2022-12-01 23:15:46 +0000 UTC (0+5 container statuses recorded) Dec 1 23:43:50.238: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:43:50.238: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:43:50.238: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:43:50.238: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:43:50.238: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:43:50.238: INFO: kube-controller-manager-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:50.238: INFO: Container kube-controller-manager ready: true, restart count 2 Dec 1 23:43:50.238: INFO: etcd-manager-events-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:50.238: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:43:50.832: INFO: Latency metrics for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:43:50.832: INFO: Logging node info for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:43:51.009: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-46-84.ap-northeast-2.compute.internal e408f2b8-c522-4e88-973a-78a2b7a2298f 23526 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-46-84.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-3465 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-46-84.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03a01a80f538d5d6b"} io.cilium.network.ipv4-cilium-host:172.20.38.72 io.cilium.network.ipv4-health-ip:172.20.48.223 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:40 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:40:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kubelet Update v1 2022-12-01 23:42:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03a01a80f538d5d6b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:40 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:43:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:43:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:43:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:43:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.46.84,},NodeAddress{Type:ExternalIP,Address:54.180.162.145,},NodeAddress{Type:Hostname,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-162-145.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec285e4d0d3397ed7e702d489e42654b,SystemUUID:ec285e4d-0d33-97ed-7e70-2d489e42654b,BootID:d6d1d15f-8121-4e40-b71c-e5b51419dc1b,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:43:51.010: INFO: Logging kubelet events for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:43:51.188: INFO: Logging pods the kubelet thinks is on node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:43:51.368: INFO: cilium-7h664 started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:43:51.369: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:43:51.369: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:43:51.369: INFO: busybox-a9ea1084-e47e-4ba9-b2ba-b1bbecaf8453 started at 2022-12-01 23:42:54 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:51.369: INFO: Container busybox ready: true, restart count 0 Dec 1 23:43:51.369: INFO: boom-server started at 2022-12-01 23:43:07 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:51.369: INFO: Container boom-server ready: true, restart count 0 Dec 1 23:43:51.369: INFO: webserver-777895d859-vz76x started at 2022-12-01 23:43:49 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:51.369: INFO: Container httpd ready: true, restart count 0 Dec 1 23:43:51.369: INFO: test-rolling-update-with-lb-7db9cf5ff8-q9gxs started at 2022-12-01 23:42:24 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:51.369: INFO: Container agnhost ready: true, restart count 0 Dec 1 23:43:51.369: INFO: webserver-847dcfb7fb-pg8td started at 2022-12-01 23:43:49 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:51.369: INFO: Container httpd ready: false, restart count 0 Dec 1 23:43:51.369: INFO: ebs-csi-node-lfxzl started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:43:51.369: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:43:51.369: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:43:51.369: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:43:51.369: INFO: pod2 started at 2022-12-01 23:43:44 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:51.369: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:43:51.964: INFO: Latency metrics for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:43:51.964: INFO: Logging node info for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:43:52.141: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-54-150.ap-northeast-2.compute.internal 2a8a1d71-d00b-4259-82bd-841f96508593 20359 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-54-150.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-54-150.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02d689db76221fe72"} io.cilium.network.ipv4-cilium-host:172.20.43.0 io.cilium.network.ipv4-health-ip:172.20.58.253 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-12-01 23:17:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:38 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:41:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02d689db76221fe72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:38 +0000 UTC,LastTransitionTime:2022-12-01 23:17:38 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:41:51 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:41:51 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:41:51 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:41:51 +0000 UTC,LastTransitionTime:2022-12-01 23:17:30 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.150,},NodeAddress{Type:ExternalIP,Address:54.180.133.96,},NodeAddress{Type:Hostname,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-133-96.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec20870a7ff0c6740650f49249abc430,SystemUUID:ec20870a-7ff0-c674-0650-f49249abc430,BootID:803a40bf-5805-4ae9-b933-fde612d75f1d,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:43:52.142: INFO: Logging kubelet events for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:43:52.321: INFO: Logging pods the kubelet thinks is on node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:43:52.501: INFO: test-rolling-update-with-lb-7db9cf5ff8-wrmt7 started at 2022-12-01 23:42:23 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:52.501: INFO: Container agnhost ready: true, restart count 0 Dec 1 23:43:52.501: INFO: webserver-777895d859-vl5qb started at 2022-12-01 23:43:49 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:52.501: INFO: Container httpd ready: true, restart count 0 Dec 1 23:43:52.501: INFO: ebs-csi-node-n5tcr started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:43:52.501: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:43:52.501: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:43:52.501: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:43:52.501: INFO: coredns-5fcc7b6498-vcsml started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:52.502: INFO: Container coredns ready: true, restart count 0 Dec 1 23:43:52.502: INFO: cilium-2bxvd started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:43:52.502: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:43:52.502: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:43:52.502: INFO: execpodqxsbr started at 2022-12-01 23:43:48 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:52.502: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:43:52.502: INFO: pod1 started at 2022-12-01 23:43:41 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:52.502: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:43:52.502: INFO: coredns-autoscaler-6658b4bf85-92nhw started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:52.502: INFO: Container autoscaler ready: true, restart count 0 Dec 1 23:43:52.502: INFO: test-pod-e3d577e5-bee1-48c4-90ba-50533ddf154a started at 2022-12-01 23:43:48 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:52.502: INFO: Container agnhost-container ready: false, restart count 0 Dec 1 23:43:54.813: INFO: Latency metrics for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:43:54.813: INFO: Logging node info for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:43:54.990: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-57.ap-northeast-2.compute.internal 2ea0d3e0-dcd9-4562-ada8-ee5e6b866480 23627 0 2022-12-01 23:17:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-57.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-55-57.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-174":"ip-172-20-55-57.ap-northeast-2.compute.internal","csi-mock-csi-mock-volumes-1611":"csi-mock-csi-mock-volumes-1611","csi-mock-csi-mock-volumes-3641":"csi-mock-csi-mock-volumes-3641","ebs.csi.aws.com":"i-02200981456309a72"} io.cilium.network.ipv4-cilium-host:172.20.51.99 io.cilium.network.ipv4-health-ip:172.20.36.120 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:44 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-12-01 23:38:39 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-01 23:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02200981456309a72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:44 +0000 UTC,LastTransitionTime:2022-12-01 23:17:44 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:43:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:43:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:43:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:43:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.57,},NodeAddress{Type:ExternalIP,Address:3.36.130.97,},NodeAddress{Type:Hostname,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-36-130-97.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bf2fd9fd22cd1aac5da0aa67f7a08,SystemUUID:ec2bf2fd-9fd2-2cd1-aac5-da0aa67f7a08,BootID:c4c37f48-1b62-417d-82bf-ea460350f712,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-936^47b47419-71d1-11ed-be23-02bdc8e24a07 kubernetes.io/csi/csi-mock-csi-mock-volumes-3641^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-936^47b47419-71d1-11ed-be23-02bdc8e24a07,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-3641^4,DevicePath:,},},Config:nil,},} Dec 1 23:43:54.990: INFO: Logging kubelet events for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:43:55.169: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:43:55.356: INFO: cilium-nwg58 started at 2022-12-01 23:17:15 +0000 UTC (1+1 container statuses recorded) Dec 1 23:43:55.357: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:43:55.357: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:43:55.357: INFO: pvc-volume-tester-jb5sw started at 2022-12-01 23:43:50 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:55.357: INFO: Container volume-tester ready: true, restart count 0 Dec 1 23:43:55.357: INFO: csi-mockplugin-resizer-0 started at 2022-12-01 23:43:26 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:55.357: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:43:55.357: INFO: pvc-volume-tester-2rtj6 started at 2022-12-01 23:43:53 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:55.357: INFO: Container volume-tester ready: false, restart count 0 Dec 1 23:43:55.357: INFO: coredns-5fcc7b6498-x9kjd started at 2022-12-01 23:17:56 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:55.357: INFO: Container coredns ready: true, restart count 0 Dec 1 23:43:55.357: INFO: csi-mockplugin-0 started at 2022-12-01 23:43:25 +0000 UTC (0+3 container statuses recorded) Dec 1 23:43:55.357: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:43:55.357: INFO: Container driver-registrar ready: true, restart count 0 Dec 1 23:43:55.357: INFO: Container mock ready: true, restart count 0 Dec 1 23:43:55.357: INFO: csi-mockplugin-attacher-0 started at 2022-12-01 23:43:26 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:55.357: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:43:55.357: INFO: startup-script started at 2022-12-01 23:43:16 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:55.357: INFO: Container startup-script ready: true, restart count 0 Dec 1 23:43:55.357: INFO: csi-mockplugin-0 started at 2022-12-01 23:43:27 +0000 UTC (0+3 container statuses recorded) Dec 1 23:43:55.357: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:43:55.357: INFO: Container driver-registrar ready: true, restart count 0 Dec 1 23:43:55.357: INFO: Container mock ready: true, restart count 0 Dec 1 23:43:55.357: INFO: ebs-csi-node-5zdr7 started at 2022-12-01 23:17:15 +0000 UTC (0+3 container statuses recorded) Dec 1 23:43:55.357: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:43:55.357: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:43:55.357: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:43:55.357: INFO: csi-hostpathplugin-0 started at 2022-12-01 23:42:33 +0000 UTC (0+7 container statuses recorded) Dec 1 23:43:55.357: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:43:55.357: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:43:55.357: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:43:55.357: INFO: Container csi-snapshotter ready: true, restart count 0 Dec 1 23:43:55.357: INFO: Container hostpath ready: true, restart count 0 Dec 1 23:43:55.357: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:43:55.357: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:43:56.071: INFO: Latency metrics for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:43:56.071: INFO: Logging node info for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:43:56.249: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-60-229.ap-northeast-2.compute.internal 63277852-d67d-402c-b963-ae3a2995d791 19833 0 2022-12-01 23:17:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-60-229.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-60-229.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0441b4e9d8ac12445"} io.cilium.network.ipv4-cilium-host:172.20.50.202 io.cilium.network.ipv4-health-ip:172.20.42.191 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:45 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:41:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-0441b4e9d8ac12445,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:45 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:41:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:41:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:41:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:41:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.229,},NodeAddress{Type:ExternalIP,Address:43.201.70.39,},NodeAddress{Type:Hostname,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-43-201-70-39.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec24f7651bea0c9073fca065a65b3820,SystemUUID:ec24f765-1bea-0c90-73fc-a065a65b3820,BootID:5386d7df-0b64-4c56-85ff-3f8a9a84c900,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:43:56.250: INFO: Logging kubelet events for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:43:56.430: INFO: Logging pods the kubelet thinks is on node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:43:56.612: INFO: hostexec-ip-172-20-60-229.ap-northeast-2.compute.internal-ddmnc started at 2022-12-01 23:43:31 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:56.612: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:43:56.612: INFO: webserver-777895d859-ns29c started at 2022-12-01 23:43:49 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:56.612: INFO: Container httpd ready: true, restart count 0 Dec 1 23:43:56.612: INFO: test-rolling-update-with-lb-7db9cf5ff8-wx9pm started at 2022-12-01 23:42:23 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:56.612: INFO: Container agnhost ready: true, restart count 0 Dec 1 23:43:56.612: INFO: ebs-csi-node-wqh44 started at 2022-12-01 23:17:12 +0000 UTC (0+3 container statuses recorded) Dec 1 23:43:56.612: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:43:56.612: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:43:56.612: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:43:56.612: INFO: cilium-5dnwd started at 2022-12-01 23:17:12 +0000 UTC (1+1 container statuses recorded) Dec 1 23:43:56.612: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:43:56.612: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:43:56.612: INFO: liveness-5539dabf-86d2-4e43-98d5-24fdb9d93672 started at 2022-12-01 23:41:31 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:56.612: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:43:56.612: INFO: hostexec-ip-172-20-60-229.ap-northeast-2.compute.internal-hhcmj started at 2022-12-01 23:43:51 +0000 UTC (0+1 container statuses recorded) Dec 1 23:43:56.612: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:43:57.235: INFO: Latency metrics for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:43:57.235: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volume-expand-9861" for this suite.
Find pod-d990c8d7-602a-45c5-8490-2510698d5907 mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(Always\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\snew\spod\sfsgroup\sapplied\sto\svolume\scontents$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Dec 2 00:02:34.579: Unexpected error: <*errors.errorString | 0xc0049de3d0>: { s: "pod \"pod-9649a4a3-3dc3-4a8c-bcd4-9a9a075b644b\" is not Running: timed out waiting for the condition", } pod "pod-9649a4a3-3dc3-4a8c-bcd4-9a9a075b644b" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250from junit_04.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Dec 1 23:57:32.423: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename fsgroupchangepolicy �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Dec 1 23:57:33.327: INFO: Creating resource for dynamic PV Dec 1 23:57:33.327: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass fsgroupchangepolicy-252-e2e-sc55b5n �[1mSTEP�[0m: creating a claim Dec 1 23:57:33.507: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-252 with fsgroup 1000 Dec 2 00:02:34.578: FAIL: Unexpected error: <*errors.errorString | 0xc0049de3d0>: { s: "pod \"pod-9649a4a3-3dc3-4a8c-bcd4-9a9a075b644b\" is not Running: timed out waiting for the condition", } pod "pod-9649a4a3-3dc3-4a8c-bcd4-9a9a075b644b" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc0027b8160, 0xc002b1b0d8, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250 +0x1c5 k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:221 +0x219 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000226480) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000226480) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000226480, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 �[1mSTEP�[0m: Deleting pvc Dec 2 00:02:34.757: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comlzr85" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "fsgroupchangepolicy-252". �[1mSTEP�[0m: Found 5 events. Dec 2 00:02:35.296: INFO: At 2022-12-01 23:57:33 +0000 UTC - event for ebs.csi.aws.comlzr85: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Dec 2 00:02:35.296: INFO: At 2022-12-01 23:57:33 +0000 UTC - event for ebs.csi.aws.comlzr85: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Dec 2 00:02:35.296: INFO: At 2022-12-01 23:57:33 +0000 UTC - event for ebs.csi.aws.comlzr85: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-252/ebs.csi.aws.comlzr85" Dec 2 00:02:35.296: INFO: At 2022-12-01 23:57:44 +0000 UTC - event for ebs.csi.aws.comlzr85: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-252-e2e-sc55b5n": rpc error: code = DeadlineExceeded desc = context deadline exceeded Dec 2 00:02:35.296: INFO: At 2022-12-02 00:02:34 +0000 UTC - event for pod-9649a4a3-3dc3-4a8c-bcd4-9a9a075b644b: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-252/ebs.csi.aws.comlzr85" Dec 2 00:02:35.474: INFO: POD NODE PHASE GRACE CONDITIONS Dec 2 00:02:35.474: INFO: pod-9649a4a3-3dc3-4a8c-bcd4-9a9a075b644b Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2022-12-02 00:02:34 +0000 UTC SchedulerError running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-252/ebs.csi.aws.comlzr85"}] Dec 2 00:02:35.474: INFO: Dec 2 00:02:35.653: INFO: Logging node info for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:02:35.832: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-156.ap-northeast-2.compute.internal e92fe5b6-ec1b-421a-992a-1182ff1e3522 48951 0 2022-12-01 23:15:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-156.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03fe1aa2fac33129f"} io.cilium.network.ipv4-cilium-host:172.20.40.211 io.cilium.network.ipv4-health-ip:172.20.46.196 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-12-01 23:15:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-12-01 23:15:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-12-01 23:15:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:16:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:21 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03fe1aa2fac33129f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:21 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:16:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.156,},NodeAddress{Type:ExternalIP,Address:15.165.75.247,},NodeAddress{Type:Hostname,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-165-75-247.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccbd8e0eb399c0115bc276d51c7ab,SystemUUID:ec2ccbd8-e0eb-399c-0115-bc276d51c7ab,BootID:4c45c6e5-af61-46fe-b037-b7cc919a53fc,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.2],SizeBytes:41958567,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.2],SizeBytes:41441104,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.2],SizeBytes:4954598,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:02:35.832: INFO: Logging kubelet events for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:02:36.013: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:02:36.196: INFO: ebs-csi-controller-764b64d679-5qsrv started at 2022-12-01 23:15:46 +0000 UTC (0+5 container statuses recorded) Dec 2 00:02:36.196: INFO: Container csi-attacher ready: true, restart count 0 Dec 2 00:02:36.196: INFO: Container csi-provisioner ready: true, restart count 0 Dec 2 00:02:36.196: INFO: Container csi-resizer ready: true, restart count 0 Dec 2 00:02:36.196: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:02:36.196: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:02:36.196: INFO: kube-controller-manager-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:36.196: INFO: Container kube-controller-manager ready: true, restart count 2 Dec 2 00:02:36.196: INFO: etcd-manager-events-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:36.196: INFO: Container etcd-manager ready: true, restart count 0 Dec 2 00:02:36.196: INFO: etcd-manager-main-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:36.196: INFO: Container etcd-manager ready: true, restart count 0 Dec 2 00:02:36.196: INFO: kops-controller-g4pqg started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:36.196: INFO: Container kops-controller ready: true, restart count 0 Dec 2 00:02:36.196: INFO: cilium-pb6vg started at 2022-12-01 23:15:46 +0000 UTC (1+1 container statuses recorded) Dec 2 00:02:36.196: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:02:36.196: INFO: Container cilium-agent ready: true, restart count 1 Dec 2 00:02:36.196: INFO: kube-scheduler-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:36.196: INFO: Container kube-scheduler ready: true, restart count 0 Dec 2 00:02:36.196: INFO: kube-apiserver-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+2 container statuses recorded) Dec 2 00:02:36.196: INFO: Container healthcheck ready: true, restart count 0 Dec 2 00:02:36.196: INFO: Container kube-apiserver ready: true, restart count 1 Dec 2 00:02:36.196: INFO: ebs-csi-node-m7cgg started at 2022-12-01 23:15:46 +0000 UTC (0+3 container statuses recorded) Dec 2 00:02:36.196: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:02:36.196: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:02:36.196: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:02:36.196: INFO: dns-controller-5fb778d9d5-qrfkv started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:36.196: INFO: Container dns-controller ready: true, restart count 0 Dec 2 00:02:36.196: INFO: cilium-operator-587dd66f87-pgs67 started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:36.196: INFO: Container cilium-operator ready: true, restart count 0 Dec 2 00:02:36.807: INFO: Latency metrics for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:02:36.807: INFO: Logging node info for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:02:36.985: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-46-84.ap-northeast-2.compute.internal e408f2b8-c522-4e88-973a-78a2b7a2298f 50362 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-46-84.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-3465 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-46-84.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03a01a80f538d5d6b"} io.cilium.network.ipv4-cilium-host:172.20.38.72 io.cilium.network.ipv4-health-ip:172.20.48.223 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:40 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:40:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kubelet Update v1 2022-12-01 23:57:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03a01a80f538d5d6b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:40 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-02 00:02:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-02 00:02:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-02 00:02:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-02 00:02:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.46.84,},NodeAddress{Type:ExternalIP,Address:54.180.162.145,},NodeAddress{Type:Hostname,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-162-145.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec285e4d0d3397ed7e702d489e42654b,SystemUUID:ec285e4d-0d33-97ed-7e70-2d489e42654b,BootID:d6d1d15f-8121-4e40-b71c-e5b51419dc1b,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:02:36.985: INFO: Logging kubelet events for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:02:37.165: INFO: Logging pods the kubelet thinks is on node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:02:37.345: INFO: cilium-7h664 started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 2 00:02:37.345: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:02:37.345: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:02:37.345: INFO: ebs-csi-node-lfxzl started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 2 00:02:37.345: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:02:37.345: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:02:37.345: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:02:37.937: INFO: Latency metrics for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:02:37.937: INFO: Logging node info for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:02:38.115: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-54-150.ap-northeast-2.compute.internal 2a8a1d71-d00b-4259-82bd-841f96508593 48413 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-54-150.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-54-150.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02d689db76221fe72"} io.cilium.network.ipv4-cilium-host:172.20.43.0 io.cilium.network.ipv4-health-ip:172.20.58.253 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-12-01 23:17:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:38 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:56:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02d689db76221fe72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:38 +0000 UTC,LastTransitionTime:2022-12-01 23:17:38 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:57:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:30 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.150,},NodeAddress{Type:ExternalIP,Address:54.180.133.96,},NodeAddress{Type:Hostname,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-133-96.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec20870a7ff0c6740650f49249abc430,SystemUUID:ec20870a-7ff0-c674-0650-f49249abc430,BootID:803a40bf-5805-4ae9-b933-fde612d75f1d,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 k8s.gcr.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:e7fddbaac4c3451da2365ab90bad149d32f11409738034e41e0f460927f7c276 k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.4],SizeBytes:24757245,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:02:38.116: INFO: Logging kubelet events for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:02:38.296: INFO: Logging pods the kubelet thinks is on node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:02:38.492: INFO: coredns-autoscaler-6658b4bf85-92nhw started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:38.492: INFO: Container autoscaler ready: true, restart count 0 Dec 2 00:02:38.492: INFO: ebs-csi-node-n5tcr started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 2 00:02:38.492: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:02:38.492: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:02:38.492: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:02:38.492: INFO: coredns-5fcc7b6498-vcsml started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:38.492: INFO: Container coredns ready: true, restart count 0 Dec 2 00:02:38.492: INFO: cilium-2bxvd started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 2 00:02:38.492: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:02:38.492: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:02:39.088: INFO: Latency metrics for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:02:39.088: INFO: Logging node info for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:02:39.266: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-57.ap-northeast-2.compute.internal 2ea0d3e0-dcd9-4562-ada8-ee5e6b866480 49228 0 2022-12-01 23:17:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-57.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-55-57.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02200981456309a72"} io.cilium.network.ipv4-cilium-host:172.20.51.99 io.cilium.network.ipv4-health-ip:172.20.36.120 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:44 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:54:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02200981456309a72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:44 +0000 UTC,LastTransitionTime:2022-12-01 23:17:44 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:57:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.57,},NodeAddress{Type:ExternalIP,Address:3.36.130.97,},NodeAddress{Type:Hostname,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-36-130-97.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bf2fd9fd22cd1aac5da0aa67f7a08,SystemUUID:ec2bf2fd-9fd2-2cd1-aac5-da0aa67f7a08,BootID:c4c37f48-1b62-417d-82bf-ea460350f712,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-936^47b47419-71d1-11ed-be23-02bdc8e24a07],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:02:39.267: INFO: Logging kubelet events for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:02:39.451: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:02:39.708: INFO: ebs-csi-node-5zdr7 started at 2022-12-01 23:17:15 +0000 UTC (0+3 container statuses recorded) Dec 2 00:02:39.708: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:02:39.708: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:02:39.708: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:02:39.708: INFO: cilium-nwg58 started at 2022-12-01 23:17:15 +0000 UTC (1+1 container statuses recorded) Dec 2 00:02:39.708: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:02:39.708: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:02:39.708: INFO: coredns-5fcc7b6498-x9kjd started at 2022-12-01 23:17:56 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:39.708: INFO: Container coredns ready: true, restart count 0 Dec 2 00:02:40.353: INFO: Latency metrics for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:02:40.353: INFO: Logging node info for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:02:40.531: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-60-229.ap-northeast-2.compute.internal 63277852-d67d-402c-b963-ae3a2995d791 49657 0 2022-12-01 23:17:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-60-229.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-60-229.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0441b4e9d8ac12445"} io.cilium.network.ipv4-cilium-host:172.20.50.202 io.cilium.network.ipv4-health-ip:172.20.42.191 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:45 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:59:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-0441b4e9d8ac12445,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:45 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.229,},NodeAddress{Type:ExternalIP,Address:43.201.70.39,},NodeAddress{Type:Hostname,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-43-201-70-39.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec24f7651bea0c9073fca065a65b3820,SystemUUID:ec24f765-1bea-0c90-73fc-a065a65b3820,BootID:5386d7df-0b64-4c56-85ff-3f8a9a84c900,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:02:40.532: INFO: Logging kubelet events for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:02:40.712: INFO: Logging pods the kubelet thinks is on node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:02:40.892: INFO: ebs-csi-node-wqh44 started at 2022-12-01 23:17:12 +0000 UTC (0+3 container statuses recorded) Dec 2 00:02:40.893: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:02:40.893: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:02:40.893: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:02:40.893: INFO: cilium-5dnwd started at 2022-12-01 23:17:12 +0000 UTC (1+1 container statuses recorded) Dec 2 00:02:40.893: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:02:40.893: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:02:41.514: INFO: Latency metrics for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:02:41.514: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "fsgroupchangepolicy-252" for this suite.
Find pod-9649a4a3-3dc3-4a8c-bcd4-9a9a075b644b mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(Always\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\svolume\scontents\sownership\schanged\sin\sfirst\spod\,\snew\spod\swith\sdifferent\sfsgroup\sapplied\sto\sthe\svolume\scontents$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Dec 2 00:01:45.645: Unexpected error: <*errors.errorString | 0xc00196f320>: { s: "pod \"pod-78d36d97-f83a-44a5-9ff0-e3ad02889249\" is not Running: timed out waiting for the condition", } pod "pod-78d36d97-f83a-44a5-9ff0-e3ad02889249" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250from junit_03.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Dec 1 23:56:43.517: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename fsgroupchangepolicy �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with different fsgroup applied to the volume contents /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Dec 1 23:56:44.401: INFO: Creating resource for dynamic PV Dec 1 23:56:44.402: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass fsgroupchangepolicy-5434-e2e-scs4wdq �[1mSTEP�[0m: creating a claim Dec 1 23:56:44.579: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-5434 with fsgroup 1000 Dec 2 00:01:45.645: FAIL: Unexpected error: <*errors.errorString | 0xc00196f320>: { s: "pod \"pod-78d36d97-f83a-44a5-9ff0-e3ad02889249\" is not Running: timed out waiting for the condition", } pod "pod-78d36d97-f83a-44a5-9ff0-e3ad02889249" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc00339fce0, 0xc0017010d8, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250 +0x1c5 k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:221 +0x219 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc0008e9380) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc0008e9380) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc0008e9380, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 �[1mSTEP�[0m: Deleting pvc Dec 2 00:01:45.822: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comhnl6v" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "fsgroupchangepolicy-5434". �[1mSTEP�[0m: Found 6 events. Dec 2 00:01:46.353: INFO: At 2022-12-01 23:56:44 +0000 UTC - event for ebs.csi.aws.comhnl6v: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Dec 2 00:01:46.353: INFO: At 2022-12-01 23:56:45 +0000 UTC - event for ebs.csi.aws.comhnl6v: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Dec 2 00:01:46.353: INFO: At 2022-12-01 23:56:45 +0000 UTC - event for ebs.csi.aws.comhnl6v: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-5434/ebs.csi.aws.comhnl6v" Dec 2 00:01:46.353: INFO: At 2022-12-01 23:56:55 +0000 UTC - event for ebs.csi.aws.comhnl6v: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-5434-e2e-scs4wdq": rpc error: code = DeadlineExceeded desc = context deadline exceeded Dec 2 00:01:46.353: INFO: At 2022-12-01 23:58:58 +0000 UTC - event for ebs.csi.aws.comhnl6v: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-5434-e2e-scs4wdq": rpc error: code = Internal desc = Could not create volume "pvc-701187ee-b47d-46a3-81e3-c346f2e91495": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Dec 2 00:01:46.353: INFO: At 2022-12-02 00:01:46 +0000 UTC - event for pod-78d36d97-f83a-44a5-9ff0-e3ad02889249: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-5434/ebs.csi.aws.comhnl6v" Dec 2 00:01:46.529: INFO: POD NODE PHASE GRACE CONDITIONS Dec 2 00:01:46.529: INFO: pod-78d36d97-f83a-44a5-9ff0-e3ad02889249 Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2022-12-02 00:01:46 +0000 UTC SchedulerError running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-5434/ebs.csi.aws.comhnl6v"}] Dec 2 00:01:46.529: INFO: Dec 2 00:01:46.707: INFO: Logging node info for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:01:46.883: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-156.ap-northeast-2.compute.internal e92fe5b6-ec1b-421a-992a-1182ff1e3522 48951 0 2022-12-01 23:15:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-156.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03fe1aa2fac33129f"} io.cilium.network.ipv4-cilium-host:172.20.40.211 io.cilium.network.ipv4-health-ip:172.20.46.196 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-12-01 23:15:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-12-01 23:15:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-12-01 23:15:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:16:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:21 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03fe1aa2fac33129f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:21 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:16:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.156,},NodeAddress{Type:ExternalIP,Address:15.165.75.247,},NodeAddress{Type:Hostname,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-165-75-247.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccbd8e0eb399c0115bc276d51c7ab,SystemUUID:ec2ccbd8-e0eb-399c-0115-bc276d51c7ab,BootID:4c45c6e5-af61-46fe-b037-b7cc919a53fc,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.2],SizeBytes:41958567,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.2],SizeBytes:41441104,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.2],SizeBytes:4954598,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:01:46.884: INFO: Logging kubelet events for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:01:47.062: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:01:47.244: INFO: ebs-csi-node-m7cgg started at 2022-12-01 23:15:46 +0000 UTC (0+3 container statuses recorded) Dec 2 00:01:47.244: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:01:47.244: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:01:47.244: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:01:47.244: INFO: dns-controller-5fb778d9d5-qrfkv started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:47.244: INFO: Container dns-controller ready: true, restart count 0 Dec 2 00:01:47.244: INFO: cilium-operator-587dd66f87-pgs67 started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:47.244: INFO: Container cilium-operator ready: true, restart count 0 Dec 2 00:01:47.244: INFO: kube-scheduler-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:47.244: INFO: Container kube-scheduler ready: true, restart count 0 Dec 2 00:01:47.244: INFO: kube-apiserver-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+2 container statuses recorded) Dec 2 00:01:47.244: INFO: Container healthcheck ready: true, restart count 0 Dec 2 00:01:47.244: INFO: Container kube-apiserver ready: true, restart count 1 Dec 2 00:01:47.244: INFO: etcd-manager-main-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:47.244: INFO: Container etcd-manager ready: true, restart count 0 Dec 2 00:01:47.244: INFO: kops-controller-g4pqg started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:47.244: INFO: Container kops-controller ready: true, restart count 0 Dec 2 00:01:47.244: INFO: cilium-pb6vg started at 2022-12-01 23:15:46 +0000 UTC (1+1 container statuses recorded) Dec 2 00:01:47.244: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:01:47.244: INFO: Container cilium-agent ready: true, restart count 1 Dec 2 00:01:47.244: INFO: ebs-csi-controller-764b64d679-5qsrv started at 2022-12-01 23:15:46 +0000 UTC (0+5 container statuses recorded) Dec 2 00:01:47.244: INFO: Container csi-attacher ready: true, restart count 0 Dec 2 00:01:47.245: INFO: Container csi-provisioner ready: true, restart count 0 Dec 2 00:01:47.245: INFO: Container csi-resizer ready: true, restart count 0 Dec 2 00:01:47.245: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:01:47.245: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:01:47.245: INFO: kube-controller-manager-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:47.245: INFO: Container kube-controller-manager ready: true, restart count 2 Dec 2 00:01:47.245: INFO: etcd-manager-events-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:47.245: INFO: Container etcd-manager ready: true, restart count 0 Dec 2 00:01:47.846: INFO: Latency metrics for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:01:47.846: INFO: Logging node info for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:01:48.022: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-46-84.ap-northeast-2.compute.internal e408f2b8-c522-4e88-973a-78a2b7a2298f 48987 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-46-84.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-3465 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-46-84.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03a01a80f538d5d6b"} io.cilium.network.ipv4-cilium-host:172.20.38.72 io.cilium.network.ipv4-health-ip:172.20.48.223 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:40 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:40:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kubelet Update v1 2022-12-01 23:57:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03a01a80f538d5d6b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:40 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:57:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.46.84,},NodeAddress{Type:ExternalIP,Address:54.180.162.145,},NodeAddress{Type:Hostname,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-162-145.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec285e4d0d3397ed7e702d489e42654b,SystemUUID:ec285e4d-0d33-97ed-7e70-2d489e42654b,BootID:d6d1d15f-8121-4e40-b71c-e5b51419dc1b,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:01:48.023: INFO: Logging kubelet events for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:01:48.201: INFO: Logging pods the kubelet thinks is on node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:01:48.439: INFO: ebs-csi-node-lfxzl started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 2 00:01:48.439: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:01:48.439: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:01:48.439: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:01:48.439: INFO: cilium-7h664 started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 2 00:01:48.439: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:01:48.439: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:01:49.038: INFO: Latency metrics for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:01:49.038: INFO: Logging node info for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:01:49.215: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-54-150.ap-northeast-2.compute.internal 2a8a1d71-d00b-4259-82bd-841f96508593 48413 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-54-150.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-54-150.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02d689db76221fe72"} io.cilium.network.ipv4-cilium-host:172.20.43.0 io.cilium.network.ipv4-health-ip:172.20.58.253 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-12-01 23:17:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:38 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:56:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02d689db76221fe72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:38 +0000 UTC,LastTransitionTime:2022-12-01 23:17:38 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:57:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:30 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.150,},NodeAddress{Type:ExternalIP,Address:54.180.133.96,},NodeAddress{Type:Hostname,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-133-96.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec20870a7ff0c6740650f49249abc430,SystemUUID:ec20870a-7ff0-c674-0650-f49249abc430,BootID:803a40bf-5805-4ae9-b933-fde612d75f1d,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 k8s.gcr.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:e7fddbaac4c3451da2365ab90bad149d32f11409738034e41e0f460927f7c276 k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.4],SizeBytes:24757245,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:01:49.216: INFO: Logging kubelet events for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:01:49.394: INFO: Logging pods the kubelet thinks is on node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:01:49.574: INFO: ebs-csi-node-n5tcr started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 2 00:01:49.574: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:01:49.574: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:01:49.574: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:01:49.574: INFO: cilium-2bxvd started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 2 00:01:49.574: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:01:49.574: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:01:49.574: INFO: coredns-5fcc7b6498-vcsml started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:49.574: INFO: Container coredns ready: true, restart count 0 Dec 2 00:01:49.574: INFO: coredns-autoscaler-6658b4bf85-92nhw started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:49.574: INFO: Container autoscaler ready: true, restart count 0 Dec 2 00:01:50.169: INFO: Latency metrics for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:01:50.169: INFO: Logging node info for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:01:50.346: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-57.ap-northeast-2.compute.internal 2ea0d3e0-dcd9-4562-ada8-ee5e6b866480 49228 0 2022-12-01 23:17:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-57.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-55-57.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02200981456309a72"} io.cilium.network.ipv4-cilium-host:172.20.51.99 io.cilium.network.ipv4-health-ip:172.20.36.120 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:44 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:54:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02200981456309a72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:44 +0000 UTC,LastTransitionTime:2022-12-01 23:17:44 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:57:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:57:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.57,},NodeAddress{Type:ExternalIP,Address:3.36.130.97,},NodeAddress{Type:Hostname,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-36-130-97.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bf2fd9fd22cd1aac5da0aa67f7a08,SystemUUID:ec2bf2fd-9fd2-2cd1-aac5-da0aa67f7a08,BootID:c4c37f48-1b62-417d-82bf-ea460350f712,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-936^47b47419-71d1-11ed-be23-02bdc8e24a07],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:01:50.347: INFO: Logging kubelet events for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:01:50.525: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:01:50.705: INFO: ebs-csi-node-5zdr7 started at 2022-12-01 23:17:15 +0000 UTC (0+3 container statuses recorded) Dec 2 00:01:50.705: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:01:50.705: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:01:50.705: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:01:50.705: INFO: cilium-nwg58 started at 2022-12-01 23:17:15 +0000 UTC (1+1 container statuses recorded) Dec 2 00:01:50.705: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:01:50.705: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:01:50.705: INFO: coredns-5fcc7b6498-x9kjd started at 2022-12-01 23:17:56 +0000 UTC (0+1 container statuses recorded) Dec 2 00:01:50.705: INFO: Container coredns ready: true, restart count 0 Dec 2 00:01:51.339: INFO: Latency metrics for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:01:51.339: INFO: Logging node info for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:01:51.516: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-60-229.ap-northeast-2.compute.internal 63277852-d67d-402c-b963-ae3a2995d791 49657 0 2022-12-01 23:17:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-60-229.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-60-229.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0441b4e9d8ac12445"} io.cilium.network.ipv4-cilium-host:172.20.50.202 io.cilium.network.ipv4-health-ip:172.20.42.191 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:45 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:59:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-0441b4e9d8ac12445,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:45 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.229,},NodeAddress{Type:ExternalIP,Address:43.201.70.39,},NodeAddress{Type:Hostname,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-43-201-70-39.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec24f7651bea0c9073fca065a65b3820,SystemUUID:ec24f765-1bea-0c90-73fc-a065a65b3820,BootID:5386d7df-0b64-4c56-85ff-3f8a9a84c900,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:01:51.516: INFO: Logging kubelet events for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:01:51.695: INFO: Logging pods the kubelet thinks is on node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:01:51.888: INFO: cilium-5dnwd started at 2022-12-01 23:17:12 +0000 UTC (1+1 container statuses recorded) Dec 2 00:01:51.888: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:01:51.888: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:01:51.888: INFO: ebs-csi-node-wqh44 started at 2022-12-01 23:17:12 +0000 UTC (0+3 container statuses recorded) Dec 2 00:01:51.888: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:01:51.888: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:01:51.888: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:01:52.474: INFO: Latency metrics for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:01:52.474: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "fsgroupchangepolicy-5434" for this suite.
Find pod-78d36d97-f83a-44a5-9ff0-e3ad02889249 mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(Always\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\svolume\scontents\sownership\schanged\sin\sfirst\spod\,\snew\spod\swith\ssame\sfsgroup\sapplied\sto\sthe\svolume\scontents$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Dec 1 23:38:05.868: Unexpected error: <*errors.errorString | 0xc0043a4cd0>: { s: "pod \"pod-291a3e6b-e8f8-48ac-adc3-f586c5dc4601\" is not Running: timed out waiting for the condition", } pod "pod-291a3e6b-e8f8-48ac-adc3-f586c5dc4601" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250from junit_17.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Dec 1 23:33:03.732: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename fsgroupchangepolicy �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup applied to the volume contents /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Dec 1 23:33:04.622: INFO: Creating resource for dynamic PV Dec 1 23:33:04.623: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass fsgroupchangepolicy-9841-e2e-scsqmrs �[1mSTEP�[0m: creating a claim Dec 1 23:33:04.802: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-9841 with fsgroup 1000 Dec 1 23:38:05.868: FAIL: Unexpected error: <*errors.errorString | 0xc0043a4cd0>: { s: "pod \"pod-291a3e6b-e8f8-48ac-adc3-f586c5dc4601\" is not Running: timed out waiting for the condition", } pod "pod-291a3e6b-e8f8-48ac-adc3-f586c5dc4601" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc003f4f1e0, 0xc0021f70d8, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250 +0x1c5 k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:221 +0x219 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000ab4180) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000ab4180) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000ab4180, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 �[1mSTEP�[0m: Deleting pvc Dec 1 23:38:06.045: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comxcsn7" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "fsgroupchangepolicy-9841". �[1mSTEP�[0m: Found 5 events. Dec 1 23:38:06.585: INFO: At 2022-12-01 23:33:04 +0000 UTC - event for ebs.csi.aws.comxcsn7: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Dec 1 23:38:06.585: INFO: At 2022-12-01 23:33:05 +0000 UTC - event for ebs.csi.aws.comxcsn7: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Dec 1 23:38:06.585: INFO: At 2022-12-01 23:33:05 +0000 UTC - event for ebs.csi.aws.comxcsn7: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-9841/ebs.csi.aws.comxcsn7" Dec 1 23:38:06.585: INFO: At 2022-12-01 23:33:15 +0000 UTC - event for ebs.csi.aws.comxcsn7: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-9841-e2e-scsqmrs": rpc error: code = DeadlineExceeded desc = context deadline exceeded Dec 1 23:38:06.585: INFO: At 2022-12-01 23:38:06 +0000 UTC - event for pod-291a3e6b-e8f8-48ac-adc3-f586c5dc4601: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-9841/ebs.csi.aws.comxcsn7" Dec 1 23:38:06.762: INFO: POD NODE PHASE GRACE CONDITIONS Dec 1 23:38:06.762: INFO: pod-291a3e6b-e8f8-48ac-adc3-f586c5dc4601 Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2022-12-01 23:38:06 +0000 UTC SchedulerError running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-9841/ebs.csi.aws.comxcsn7"}] Dec 1 23:38:06.762: INFO: Dec 1 23:38:06.940: INFO: Logging node info for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:38:07.117: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-156.ap-northeast-2.compute.internal e92fe5b6-ec1b-421a-992a-1182ff1e3522 12007 0 2022-12-01 23:15:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-156.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03fe1aa2fac33129f"} io.cilium.network.ipv4-cilium-host:172.20.40.211 io.cilium.network.ipv4-health-ip:172.20.46.196 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-12-01 23:15:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-12-01 23:15:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-12-01 23:15:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:16:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:21 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03fe1aa2fac33129f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:21 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:38:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:38:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:38:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:38:02 +0000 UTC,LastTransitionTime:2022-12-01 23:16:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.156,},NodeAddress{Type:ExternalIP,Address:15.165.75.247,},NodeAddress{Type:Hostname,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-165-75-247.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccbd8e0eb399c0115bc276d51c7ab,SystemUUID:ec2ccbd8-e0eb-399c-0115-bc276d51c7ab,BootID:4c45c6e5-af61-46fe-b037-b7cc919a53fc,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.2],SizeBytes:41958567,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.2],SizeBytes:41441104,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.2],SizeBytes:4954598,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:38:07.117: INFO: Logging kubelet events for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:38:07.296: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:38:07.479: INFO: kube-controller-manager-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:07.480: INFO: Container kube-controller-manager ready: true, restart count 2 Dec 1 23:38:07.480: INFO: etcd-manager-events-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:07.480: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:38:07.480: INFO: etcd-manager-main-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:07.480: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:38:07.480: INFO: kops-controller-g4pqg started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:07.480: INFO: Container kops-controller ready: true, restart count 0 Dec 1 23:38:07.480: INFO: cilium-pb6vg started at 2022-12-01 23:15:46 +0000 UTC (1+1 container statuses recorded) Dec 1 23:38:07.480: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:38:07.480: INFO: Container cilium-agent ready: true, restart count 1 Dec 1 23:38:07.480: INFO: ebs-csi-controller-764b64d679-5qsrv started at 2022-12-01 23:15:46 +0000 UTC (0+5 container statuses recorded) Dec 1 23:38:07.480: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:38:07.480: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:38:07.480: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:38:07.480: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:38:07.480: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:38:07.480: INFO: kube-scheduler-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:07.480: INFO: Container kube-scheduler ready: true, restart count 0 Dec 1 23:38:07.480: INFO: kube-apiserver-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+2 container statuses recorded) Dec 1 23:38:07.480: INFO: Container healthcheck ready: true, restart count 0 Dec 1 23:38:07.480: INFO: Container kube-apiserver ready: true, restart count 1 Dec 1 23:38:07.480: INFO: ebs-csi-node-m7cgg started at 2022-12-01 23:15:46 +0000 UTC (0+3 container statuses recorded) Dec 1 23:38:07.480: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:38:07.480: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:38:07.480: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:38:07.480: INFO: dns-controller-5fb778d9d5-qrfkv started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:07.480: INFO: Container dns-controller ready: true, restart count 0 Dec 1 23:38:07.480: INFO: cilium-operator-587dd66f87-pgs67 started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:07.480: INFO: Container cilium-operator ready: true, restart count 0 Dec 1 23:38:08.065: INFO: Latency metrics for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:38:08.065: INFO: Logging node info for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:38:08.242: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-46-84.ap-northeast-2.compute.internal e408f2b8-c522-4e88-973a-78a2b7a2298f 9955 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-46-84.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-3911":"csi-mock-csi-mock-volumes-3911","csi-mock-csi-mock-volumes-8116":"csi-mock-csi-mock-volumes-8116","ebs.csi.aws.com":"i-03a01a80f538d5d6b"} io.cilium.network.ipv4-cilium-host:172.20.38.72 io.cilium.network.ipv4-health-ip:172.20.48.223 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:40 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:21:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2022-12-01 23:33:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03a01a80f538d5d6b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:40 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:33:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:33:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:33:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:33:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.46.84,},NodeAddress{Type:ExternalIP,Address:54.180.162.145,},NodeAddress{Type:Hostname,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-162-145.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec285e4d0d3397ed7e702d489e42654b,SystemUUID:ec285e4d-0d33-97ed-7e70-2d489e42654b,BootID:d6d1d15f-8121-4e40-b71c-e5b51419dc1b,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-8116^4],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:38:08.242: INFO: Logging kubelet events for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:38:08.422: INFO: Logging pods the kubelet thinks is on node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:38:08.606: INFO: busybox-readonly-false-8fc9343d-784b-461f-a297-f02198cde385 started at 2022-12-01 23:37:38 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container busybox-readonly-false-8fc9343d-784b-461f-a297-f02198cde385 ready: false, restart count 0 Dec 1 23:38:08.606: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-6rhs5 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:08.606: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-2k8kk started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:08.606: INFO: downwardapi-volume-2240ee64-81d8-49f1-92c2-98edfc5938d1 started at 2022-12-01 23:38:02 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container client-container ready: false, restart count 0 Dec 1 23:38:08.606: INFO: csi-mockplugin-0 started at 2022-12-01 23:32:29 +0000 UTC (0+3 container statuses recorded) Dec 1 23:38:08.606: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:38:08.606: INFO: Container driver-registrar ready: true, restart count 0 Dec 1 23:38:08.606: INFO: Container mock ready: true, restart count 0 Dec 1 23:38:08.606: INFO: csi-hostpathplugin-0 started at 2022-12-01 23:37:22 +0000 UTC (0+7 container statuses recorded) Dec 1 23:38:08.606: INFO: Container csi-attacher ready: false, restart count 0 Dec 1 23:38:08.606: INFO: Container csi-provisioner ready: false, restart count 0 Dec 1 23:38:08.606: INFO: Container csi-resizer ready: false, restart count 0 Dec 1 23:38:08.606: INFO: Container csi-snapshotter ready: false, restart count 0 Dec 1 23:38:08.606: INFO: Container hostpath ready: false, restart count 0 Dec 1 23:38:08.606: INFO: Container liveness-probe ready: false, restart count 0 Dec 1 23:38:08.606: INFO: Container node-driver-registrar ready: false, restart count 0 Dec 1 23:38:08.606: INFO: ebs-csi-node-lfxzl started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:38:08.606: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:38:08.606: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:38:08.606: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:38:08.606: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-gqqcm started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:08.606: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-kdzjv started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:08.606: INFO: pod-configmaps-13546401-b15e-45be-9e8a-69f7ad7500d6 started at 2022-12-01 23:38:01 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container agnhost-container ready: false, restart count 0 Dec 1 23:38:08.606: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-stdpc started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:08.606: INFO: cilium-7h664 started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:38:08.606: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:38:08.606: INFO: pvc-volume-tester-5psrl started at 2022-12-01 23:33:13 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container volume-tester ready: false, restart count 0 Dec 1 23:38:08.606: INFO: pod-6ce46920-e19d-41ad-a0e9-4aa6f7d7d00f started at 2022-12-01 23:35:08 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container test-container ready: false, restart count 0 Dec 1 23:38:08.606: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-wmskh started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:08.606: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-lwxkw started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:08.606: INFO: pod1eb14cb2-24d6-40da-86ae-15b8942808b7 started at 2022-12-01 23:37:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container nginx ready: false, restart count 0 Dec 1 23:38:08.606: INFO: csi-mockplugin-0 started at 2022-12-01 23:33:05 +0000 UTC (0+3 container statuses recorded) Dec 1 23:38:08.606: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:38:08.606: INFO: Container driver-registrar ready: true, restart count 0 Dec 1 23:38:08.606: INFO: Container mock ready: true, restart count 0 Dec 1 23:38:08.606: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-m9cn6 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:08.606: INFO: httpd started at 2022-12-01 23:37:48 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container httpd ready: false, restart count 0 Dec 1 23:38:08.606: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-4t8x8 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:08.606: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-zdrcp started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:08.606: INFO: csi-mockplugin-attacher-0 started at 2022-12-01 23:32:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container csi-attacher ready: false, restart count 0 Dec 1 23:38:08.606: INFO: csi-mockplugin-resizer-0 started at 2022-12-01 23:33:05 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container csi-resizer ready: false, restart count 0 Dec 1 23:38:08.606: INFO: pvc-volume-tester-28wlm started at 2022-12-01 23:33:12 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:08.606: INFO: Container volume-tester ready: false, restart count 0 Dec 1 23:38:09.217: INFO: Latency metrics for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:38:09.217: INFO: Logging node info for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:38:09.394: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-54-150.ap-northeast-2.compute.internal 2a8a1d71-d00b-4259-82bd-841f96508593 11959 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-54-150.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-54-150.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02d689db76221fe72"} io.cilium.network.ipv4-cilium-host:172.20.43.0 io.cilium.network.ipv4-health-ip:172.20.58.253 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-12-01 23:17:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:38 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:21:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2022-12-01 23:37:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02d689db76221fe72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:38 +0000 UTC,LastTransitionTime:2022-12-01 23:17:38 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:37:51 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:37:51 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:37:51 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:37:51 +0000 UTC,LastTransitionTime:2022-12-01 23:17:30 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.150,},NodeAddress{Type:ExternalIP,Address:54.180.133.96,},NodeAddress{Type:Hostname,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-133-96.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec20870a7ff0c6740650f49249abc430,SystemUUID:ec20870a-7ff0-c674-0650-f49249abc430,BootID:803a40bf-5805-4ae9-b933-fde612d75f1d,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:38:09.395: INFO: Logging kubelet events for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:38:09.575: INFO: Logging pods the kubelet thinks is on node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:38:09.757: INFO: ebs-csi-node-n5tcr started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:38:09.757: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:38:09.757: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:38:09.757: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:38:09.757: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-kq9dz started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:09.757: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:09.757: INFO: pod-test started at 2022-12-01 23:33:10 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:09.757: INFO: Container pod-test ready: false, restart count 0 Dec 1 23:38:09.757: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-r5sld started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:09.757: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:09.757: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-5xvz5 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:09.757: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:09.757: INFO: cilium-2bxvd started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:38:09.757: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:38:09.757: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:38:09.757: INFO: coredns-5fcc7b6498-vcsml started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:09.757: INFO: Container coredns ready: true, restart count 0 Dec 1 23:38:09.757: INFO: affinity-nodeport-w27b5 started at 2022-12-01 23:32:37 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:09.757: INFO: Container affinity-nodeport ready: false, restart count 0 Dec 1 23:38:09.757: INFO: coredns-autoscaler-6658b4bf85-92nhw started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:09.757: INFO: Container autoscaler ready: true, restart count 0 Dec 1 23:38:09.757: INFO: sysctl-7225ea1d-c155-4eac-9a6e-874c1114563d started at 2022-12-01 23:38:04 +0000 UTC (0+0 container statuses recorded) Dec 1 23:38:09.757: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-j8c74 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:09.757: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:09.757: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-7qcv7 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:09.757: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:09.757: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-6jcw4 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:09.757: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:09.757: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-htn5g started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:09.757: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:38:09.757: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-2kzlf started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:09.757: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:09.757: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-5d7kh started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:09.757: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:09.757: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-6nhjl started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:09.757: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:10.371: INFO: Latency metrics for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:38:10.371: INFO: Logging node info for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:38:10.548: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-57.ap-northeast-2.compute.internal 2ea0d3e0-dcd9-4562-ada8-ee5e6b866480 11800 0 2022-12-01 23:17:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-57.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-55-57.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02200981456309a72"} io.cilium.network.ipv4-cilium-host:172.20.51.99 io.cilium.network.ipv4-health-ip:172.20.36.120 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:44 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:21:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2022-12-01 23:32:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02200981456309a72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:44 +0000 UTC,LastTransitionTime:2022-12-01 23:17:44 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:37:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:37:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:37:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:37:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.57,},NodeAddress{Type:ExternalIP,Address:3.36.130.97,},NodeAddress{Type:Hostname,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-36-130-97.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bf2fd9fd22cd1aac5da0aa67f7a08,SystemUUID:ec2bf2fd-9fd2-2cd1-aac5-da0aa67f7a08,BootID:c4c37f48-1b62-417d-82bf-ea460350f712,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:38:10.549: INFO: Logging kubelet events for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:38:10.728: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:38:10.911: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-ddtfs started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:10.911: INFO: hostexec-ip-172-20-55-57.ap-northeast-2.compute.internal-hxmxc started at 2022-12-01 23:32:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:38:10.911: INFO: pod-0 started at 2022-12-01 23:32:51 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container donothing ready: false, restart count 0 Dec 1 23:38:10.911: INFO: csi-mockplugin-attacher-0 started at 2022-12-01 23:32:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container csi-attacher ready: false, restart count 0 Dec 1 23:38:10.911: INFO: cilium-nwg58 started at 2022-12-01 23:17:15 +0000 UTC (1+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:38:10.911: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:38:10.911: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-8l4bh started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:10.911: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-p8zf2 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:10.911: INFO: coredns-5fcc7b6498-x9kjd started at 2022-12-01 23:17:56 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container coredns ready: true, restart count 0 Dec 1 23:38:10.911: INFO: csi-mockplugin-0 started at 2022-12-01 23:32:30 +0000 UTC (0+3 container statuses recorded) Dec 1 23:38:10.911: INFO: Container csi-provisioner ready: false, restart count 0 Dec 1 23:38:10.911: INFO: Container driver-registrar ready: false, restart count 0 Dec 1 23:38:10.911: INFO: Container mock ready: false, restart count 0 Dec 1 23:38:10.911: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-nhwvz started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:10.911: INFO: webserver-pod started at 2022-12-01 23:37:54 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container agnhost ready: false, restart count 0 Dec 1 23:38:10.911: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-fmm4p started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:10.911: INFO: affinity-nodeport-mtzhz started at 2022-12-01 23:32:37 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container affinity-nodeport ready: true, restart count 0 Dec 1 23:38:10.911: INFO: pod-1 started at 2022-12-01 23:32:51 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container donothing ready: false, restart count 0 Dec 1 23:38:10.911: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-8gc5h started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:10.911: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-5z2kc started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:10.911: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-nbwwt started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:10.911: INFO: hostexec-ip-172-20-55-57.ap-northeast-2.compute.internal-vld7l started at 2022-12-01 23:38:03 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:38:10.911: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-wzpt2 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:10.911: INFO: ebs-csi-node-5zdr7 started at 2022-12-01 23:17:15 +0000 UTC (0+3 container statuses recorded) Dec 1 23:38:10.911: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:38:10.911: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:38:10.911: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:38:10.911: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-vdxsm started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:10.911: INFO: downwardapi-volume-f24841f4-c949-4672-80d9-b01bbd552efb started at 2022-12-01 23:33:12 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:10.911: INFO: Container client-container ready: false, restart count 0 Dec 1 23:38:11.514: INFO: Latency metrics for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:38:11.515: INFO: Logging node info for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:38:11.692: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-60-229.ap-northeast-2.compute.internal 63277852-d67d-402c-b963-ae3a2995d791 9853 0 2022-12-01 23:17:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-60-229.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-60-229.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-8772":"ip-172-20-60-229.ap-northeast-2.compute.internal","ebs.csi.aws.com":"i-0441b4e9d8ac12445"} io.cilium.network.ipv4-cilium-host:172.20.50.202 io.cilium.network.ipv4-health-ip:172.20.42.191 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:45 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:21:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2022-12-01 23:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-0441b4e9d8ac12445,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:45 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:33:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:33:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:33:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:33:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.229,},NodeAddress{Type:ExternalIP,Address:43.201.70.39,},NodeAddress{Type:Hostname,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-43-201-70-39.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec24f7651bea0c9073fca065a65b3820,SystemUUID:ec24f765-1bea-0c90-73fc-a065a65b3820,BootID:5386d7df-0b64-4c56-85ff-3f8a9a84c900,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:38:11.692: INFO: Logging kubelet events for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:38:11.871: INFO: Logging pods the kubelet thinks is on node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:38:12.053: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-szrp5 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:12.053: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:12.053: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-fczc8 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:12.054: INFO: pod-prestop-hook-6ce30e59-a888-45d5-9965-fd65c3006637 started at 2022-12-01 23:36:39 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Container nginx ready: false, restart count 0 Dec 1 23:38:12.054: INFO: inline-volume-tester-mk6sp started at 2022-12-01 23:29:14 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Container csi-volume-tester ready: false, restart count 0 Dec 1 23:38:12.054: INFO: pfpod started at 2022-12-01 23:36:31 +0000 UTC (0+2 container statuses recorded) Dec 1 23:38:12.054: INFO: Container portforwardtester ready: false, restart count 0 Dec 1 23:38:12.054: INFO: Container readiness ready: false, restart count 0 Dec 1 23:38:12.054: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-nr4j5 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:12.054: INFO: ebs-csi-node-wqh44 started at 2022-12-01 23:17:12 +0000 UTC (0+3 container statuses recorded) Dec 1 23:38:12.054: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:38:12.054: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:38:12.054: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:38:12.054: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-c4nx7 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:12.054: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-gvcsf started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:12.054: INFO: cilium-5dnwd started at 2022-12-01 23:17:12 +0000 UTC (1+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:38:12.054: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:38:12.054: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-4p7lw started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:12.054: INFO: affinity-nodeport-6l5kc started at 2022-12-01 23:32:37 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Container affinity-nodeport ready: true, restart count 0 Dec 1 23:38:12.054: INFO: csi-hostpathplugin-0 started at 2022-12-01 23:29:14 +0000 UTC (0+7 container statuses recorded) Dec 1 23:38:12.054: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:38:12.054: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:38:12.054: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:38:12.054: INFO: Container csi-snapshotter ready: true, restart count 0 Dec 1 23:38:12.054: INFO: Container hostpath ready: true, restart count 0 Dec 1 23:38:12.054: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:38:12.054: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:38:12.054: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-g98kz started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:12.054: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-7vclf started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:12.054: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-2hlfg started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:12.054: INFO: pod-f0a80389-9b6e-4ec3-a313-ab71f5a10ed9 started at 2022-12-01 23:33:19 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Container write-pod ready: false, restart count 0 Dec 1 23:38:12.054: INFO: pod-init-844d5754-f8c1-49cf-9b11-9a4b07203143 started at 2022-12-01 23:37:06 +0000 UTC (2+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Init container init1 ready: false, restart count 0 Dec 1 23:38:12.054: INFO: Init container init2 ready: false, restart count 0 Dec 1 23:38:12.054: INFO: Container run1 ready: false, restart count 0 Dec 1 23:38:12.054: INFO: hostexec-ip-172-20-60-229.ap-northeast-2.compute.internal-lfzj5 started at 2022-12-01 23:33:14 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:38:12.054: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-zvfn4 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:38:12.054: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:38:12.697: INFO: Latency metrics for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:38:12.697: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "fsgroupchangepolicy-9841" for this suite.
Find pod-291a3e6b-e8f8-48ac-adc3-f586c5dc4601 mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(OnRootMismatch\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\snew\spod\sfsgroup\sapplied\sto\svolume\scontents$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Dec 1 23:27:05.237: Unexpected error: <*errors.errorString | 0xc002e5d120>: { s: "pod \"pod-67758a14-58b2-42e1-97fe-7d875edc1629\" is not Running: timed out waiting for the condition", } pod "pod-67758a14-58b2-42e1-97fe-7d875edc1629" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250from junit_20.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Dec 1 23:22:03.118: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename fsgroupchangepolicy �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Dec 1 23:22:03.997: INFO: Creating resource for dynamic PV Dec 1 23:22:03.997: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass fsgroupchangepolicy-4071-e2e-sc4p2n7 �[1mSTEP�[0m: creating a claim Dec 1 23:22:04.175: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-4071 with fsgroup 1000 Dec 1 23:27:05.237: FAIL: Unexpected error: <*errors.errorString | 0xc002e5d120>: { s: "pod \"pod-67758a14-58b2-42e1-97fe-7d875edc1629\" is not Running: timed out waiting for the condition", } pod "pod-67758a14-58b2-42e1-97fe-7d875edc1629" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc001f54160, 0xc001c7f0d8, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250 +0x1c5 k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:221 +0x219 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000323e00) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000323e00) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000323e00, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 �[1mSTEP�[0m: Deleting pvc Dec 1 23:27:05.414: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comdnw6j" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "fsgroupchangepolicy-4071". �[1mSTEP�[0m: Found 6 events. Dec 1 23:27:05.945: INFO: At 2022-12-01 23:22:04 +0000 UTC - event for ebs.csi.aws.comdnw6j: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Dec 1 23:27:05.945: INFO: At 2022-12-01 23:22:04 +0000 UTC - event for ebs.csi.aws.comdnw6j: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Dec 1 23:27:05.945: INFO: At 2022-12-01 23:22:04 +0000 UTC - event for ebs.csi.aws.comdnw6j: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-4071/ebs.csi.aws.comdnw6j" Dec 1 23:27:05.945: INFO: At 2022-12-01 23:22:14 +0000 UTC - event for ebs.csi.aws.comdnw6j: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-4071-e2e-sc4p2n7": rpc error: code = DeadlineExceeded desc = context deadline exceeded Dec 1 23:27:05.945: INFO: At 2022-12-01 23:23:09 +0000 UTC - event for ebs.csi.aws.comdnw6j: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-4071-e2e-sc4p2n7": rpc error: code = Internal desc = Could not create volume "pvc-59867c04-4822-470a-a9a9-8d88b6cecba5": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Dec 1 23:27:05.945: INFO: At 2022-12-01 23:27:05 +0000 UTC - event for pod-67758a14-58b2-42e1-97fe-7d875edc1629: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-4071/ebs.csi.aws.comdnw6j" Dec 1 23:27:06.121: INFO: POD NODE PHASE GRACE CONDITIONS Dec 1 23:27:06.121: INFO: pod-67758a14-58b2-42e1-97fe-7d875edc1629 Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2022-12-01 23:27:05 +0000 UTC SchedulerError running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-4071/ebs.csi.aws.comdnw6j"}] Dec 1 23:27:06.121: INFO: Dec 1 23:27:06.298: INFO: Logging node info for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:27:06.475: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-156.ap-northeast-2.compute.internal e92fe5b6-ec1b-421a-992a-1182ff1e3522 5601 0 2022-12-01 23:15:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-156.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03fe1aa2fac33129f"} io.cilium.network.ipv4-cilium-host:172.20.40.211 io.cilium.network.ipv4-health-ip:172.20.46.196 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-12-01 23:15:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-12-01 23:15:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-12-01 23:15:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:16:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:21 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03fe1aa2fac33129f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:21 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:23:01 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:23:01 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:23:01 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:23:01 +0000 UTC,LastTransitionTime:2022-12-01 23:16:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.156,},NodeAddress{Type:ExternalIP,Address:15.165.75.247,},NodeAddress{Type:Hostname,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-165-75-247.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccbd8e0eb399c0115bc276d51c7ab,SystemUUID:ec2ccbd8-e0eb-399c-0115-bc276d51c7ab,BootID:4c45c6e5-af61-46fe-b037-b7cc919a53fc,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.2],SizeBytes:41958567,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.2],SizeBytes:41441104,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.2],SizeBytes:4954598,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:27:06.475: INFO: Logging kubelet events for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:27:06.654: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:27:07.016: INFO: kube-controller-manager-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:07.016: INFO: Container kube-controller-manager ready: true, restart count 2 Dec 1 23:27:07.016: INFO: etcd-manager-events-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:07.016: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:27:07.016: INFO: etcd-manager-main-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:07.016: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:27:07.016: INFO: kops-controller-g4pqg started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:07.016: INFO: Container kops-controller ready: true, restart count 0 Dec 1 23:27:07.016: INFO: cilium-pb6vg started at 2022-12-01 23:15:46 +0000 UTC (1+1 container statuses recorded) Dec 1 23:27:07.016: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:27:07.016: INFO: Container cilium-agent ready: true, restart count 1 Dec 1 23:27:07.016: INFO: ebs-csi-controller-764b64d679-5qsrv started at 2022-12-01 23:15:46 +0000 UTC (0+5 container statuses recorded) Dec 1 23:27:07.016: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:27:07.016: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:27:07.016: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:27:07.016: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:27:07.016: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:27:07.016: INFO: kube-scheduler-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:07.016: INFO: Container kube-scheduler ready: true, restart count 0 Dec 1 23:27:07.016: INFO: kube-apiserver-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+2 container statuses recorded) Dec 1 23:27:07.016: INFO: Container healthcheck ready: true, restart count 0 Dec 1 23:27:07.016: INFO: Container kube-apiserver ready: true, restart count 1 Dec 1 23:27:07.016: INFO: ebs-csi-node-m7cgg started at 2022-12-01 23:15:46 +0000 UTC (0+3 container statuses recorded) Dec 1 23:27:07.016: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:27:07.016: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:27:07.016: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:27:07.016: INFO: dns-controller-5fb778d9d5-qrfkv started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:07.016: INFO: Container dns-controller ready: true, restart count 0 Dec 1 23:27:07.016: INFO: cilium-operator-587dd66f87-pgs67 started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:07.016: INFO: Container cilium-operator ready: true, restart count 0 Dec 1 23:27:07.596: INFO: Latency metrics for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:27:07.596: INFO: Logging node info for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:27:07.773: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-46-84.ap-northeast-2.compute.internal e408f2b8-c522-4e88-973a-78a2b7a2298f 5114 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-46-84.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03a01a80f538d5d6b"} io.cilium.network.ipv4-cilium-host:172.20.38.72 io.cilium.network.ipv4-health-ip:172.20.48.223 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:40 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:18:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {e2e.test Update v1 2022-12-01 23:21:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03a01a80f538d5d6b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:40 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.46.84,},NodeAddress{Type:ExternalIP,Address:54.180.162.145,},NodeAddress{Type:Hostname,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-162-145.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec285e4d0d3397ed7e702d489e42654b,SystemUUID:ec285e4d-0d33-97ed-7e70-2d489e42654b,BootID:d6d1d15f-8121-4e40-b71c-e5b51419dc1b,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:27:07.773: INFO: Logging kubelet events for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:27:07.955: INFO: Logging pods the kubelet thinks is on node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:27:08.138: INFO: netserver-0 started at 2022-12-01 23:22:25 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.138: INFO: Container webserver ready: true, restart count 0 Dec 1 23:27:08.138: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-wmskh started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.138: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:08.138: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-lwxkw started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.138: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:27:08.138: INFO: webserver-deployment-847dcfb7fb-ng82l started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.138: INFO: Container httpd ready: false, restart count 0 Dec 1 23:27:08.138: INFO: webserver-deployment-847dcfb7fb-x4hj5 started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.138: INFO: Container httpd ready: false, restart count 0 Dec 1 23:27:08.138: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-m9cn6 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.138: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:08.138: INFO: pod-subpath-test-preprovisionedpv-pr5s started at 2022-12-01 23:23:47 +0000 UTC (0+2 container statuses recorded) Dec 1 23:27:08.138: INFO: Container test-container-subpath-preprovisionedpv-pr5s ready: false, restart count 0 Dec 1 23:27:08.138: INFO: Container test-container-volume-preprovisionedpv-pr5s ready: false, restart count 0 Dec 1 23:27:08.138: INFO: pod-handle-http-request started at 2022-12-01 23:27:03 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.138: INFO: Container agnhost-container ready: false, restart count 0 Dec 1 23:27:08.138: INFO: webserver-deployment-847dcfb7fb-hch2v started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.138: INFO: Container httpd ready: false, restart count 0 Dec 1 23:27:08.138: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-4t8x8 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.138: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:08.138: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-zdrcp started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.138: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:08.138: INFO: pod-c65bf730-a2f5-41c7-b67d-7e37f8ac5796 started at 2022-12-01 23:22:32 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.138: INFO: Container write-pod ready: false, restart count 0 Dec 1 23:27:08.138: INFO: netserver-0 started at 2022-12-01 23:21:41 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.138: INFO: Container webserver ready: true, restart count 0 Dec 1 23:27:08.138: INFO: hostexec-ip-172-20-46-84.ap-northeast-2.compute.internal-b2j6s started at 2022-12-01 23:22:14 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.138: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:27:08.138: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-6rhs5 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.139: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:08.139: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-2k8kk started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.139: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:27:08.139: INFO: ebs-csi-node-lfxzl started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:27:08.139: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:27:08.139: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:27:08.139: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:27:08.139: INFO: pod-projected-configmaps-fa06ceca-a667-406f-b749-dd3944e4a10a started at 2022-12-01 23:26:53 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.139: INFO: Container projected-configmap-volume-test ready: false, restart count 0 Dec 1 23:27:08.139: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-gqqcm started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.139: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:08.139: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-kdzjv started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.139: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:08.139: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-stdpc started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.139: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:08.139: INFO: cilium-7h664 started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:27:08.139: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:27:08.139: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:27:08.139: INFO: hostexec-ip-172-20-46-84.ap-northeast-2.compute.internal-5rqtf started at 2022-12-01 23:23:40 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.139: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:27:08.139: INFO: netserver-0 started at 2022-12-01 23:22:07 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.139: INFO: Container webserver ready: true, restart count 0 Dec 1 23:27:08.139: INFO: pod-secrets-fdb62779-bc6a-48f8-9e7f-782248465bb8 started at 2022-12-01 23:27:04 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:08.139: INFO: Container secret-volume-test ready: false, restart count 0 Dec 1 23:27:08.743: INFO: Latency metrics for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:27:08.743: INFO: Logging node info for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:27:08.919: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-54-150.ap-northeast-2.compute.internal 2a8a1d71-d00b-4259-82bd-841f96508593 5111 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-54-150.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-54-150.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-3080":"ip-172-20-54-150.ap-northeast-2.compute.internal","ebs.csi.aws.com":"i-02d689db76221fe72"} io.cilium.network.ipv4-cilium-host:172.20.43.0 io.cilium.network.ipv4-health-ip:172.20.58.253 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-12-01 23:17:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:38 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:21:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:22:00 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-01 23:22:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02d689db76221fe72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:38 +0000 UTC,LastTransitionTime:2022-12-01 23:17:38 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:22:10 +0000 UTC,LastTransitionTime:2022-12-01 23:17:30 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.150,},NodeAddress{Type:ExternalIP,Address:54.180.133.96,},NodeAddress{Type:Hostname,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-133-96.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec20870a7ff0c6740650f49249abc430,SystemUUID:ec20870a-7ff0-c674-0650-f49249abc430,BootID:803a40bf-5805-4ae9-b933-fde612d75f1d,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-3080^f43ea0ef-71ce-11ed-92c1-0628feb04e56 kubernetes.io/csi/csi-hostpath-ephemeral-3080^f800f595-71ce-11ed-92c1-0628feb04e56],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-3080^f43ea0ef-71ce-11ed-92c1-0628feb04e56,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-3080^f800f595-71ce-11ed-92c1-0628feb04e56,DevicePath:,},},Config:nil,},} Dec 1 23:27:08.920: INFO: Logging kubelet events for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:27:09.101: INFO: Logging pods the kubelet thinks is on node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:27:09.284: INFO: webserver-deployment-847dcfb7fb-p9cpm started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container httpd ready: false, restart count 0 Dec 1 23:27:09.284: INFO: webserver-deployment-847dcfb7fb-qb48w started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container httpd ready: false, restart count 0 Dec 1 23:27:09.284: INFO: ebs-csi-node-n5tcr started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:27:09.284: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:27:09.284: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:27:09.284: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:27:09.284: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-kq9dz started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:09.284: INFO: inline-volume-tester2-86kbx started at 2022-12-01 23:22:06 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container csi-volume-tester ready: false, restart count 0 Dec 1 23:27:09.284: INFO: netserver-1 started at 2022-12-01 23:22:25 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container webserver ready: false, restart count 0 Dec 1 23:27:09.284: INFO: netserver-1 started at 2022-12-01 23:21:41 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container webserver ready: true, restart count 0 Dec 1 23:27:09.284: INFO: csi-hostpathplugin-0 started at 2022-12-01 23:21:42 +0000 UTC (0+7 container statuses recorded) Dec 1 23:27:09.284: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:27:09.284: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:27:09.284: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:27:09.284: INFO: Container csi-snapshotter ready: true, restart count 0 Dec 1 23:27:09.284: INFO: Container hostpath ready: true, restart count 0 Dec 1 23:27:09.284: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:27:09.284: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:27:09.284: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-r5sld started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:09.284: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-5xvz5 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:27:09.284: INFO: cilium-2bxvd started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:27:09.284: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:27:09.284: INFO: coredns-5fcc7b6498-vcsml started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container coredns ready: true, restart count 0 Dec 1 23:27:09.284: INFO: exec-volume-test-preprovisionedpv-994q started at 2022-12-01 23:26:17 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container exec-container-preprovisionedpv-994q ready: false, restart count 0 Dec 1 23:27:09.284: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-j8c74 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:09.284: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-7qcv7 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:09.284: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-6jcw4 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:27:09.284: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-htn5g started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:27:09.284: INFO: netserver-1 started at 2022-12-01 23:22:07 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container webserver ready: false, restart count 0 Dec 1 23:27:09.284: INFO: hostexec-ip-172-20-54-150.ap-northeast-2.compute.internal-gjczv started at 2022-12-01 23:25:59 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:27:09.284: INFO: coredns-autoscaler-6658b4bf85-92nhw started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container autoscaler ready: true, restart count 0 Dec 1 23:27:09.284: INFO: inline-volume-tester-zzzxb started at 2022-12-01 23:21:59 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container csi-volume-tester ready: true, restart count 0 Dec 1 23:27:09.284: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-6nhjl started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:09.284: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-2kzlf started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:09.284: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-5d7kh started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:09.284: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:09.919: INFO: Latency metrics for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:27:09.919: INFO: Logging node info for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:27:10.095: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-57.ap-northeast-2.compute.internal 2ea0d3e0-dcd9-4562-ada8-ee5e6b866480 5190 0 2022-12-01 23:17:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-57.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-6520":"csi-mock-csi-mock-volumes-6520","ebs.csi.aws.com":"i-02200981456309a72"} io.cilium.network.ipv4-cilium-host:172.20.51.99 io.cilium.network.ipv4-health-ip:172.20.36.120 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:44 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:21:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:22:06 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-01 23:22:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02200981456309a72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:44 +0000 UTC,LastTransitionTime:2022-12-01 23:17:44 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:14 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:14 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:14 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:22:14 +0000 UTC,LastTransitionTime:2022-12-01 23:17:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.57,},NodeAddress{Type:ExternalIP,Address:3.36.130.97,},NodeAddress{Type:Hostname,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-36-130-97.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bf2fd9fd22cd1aac5da0aa67f7a08,SystemUUID:ec2bf2fd-9fd2-2cd1-aac5-da0aa67f7a08,BootID:c4c37f48-1b62-417d-82bf-ea460350f712,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-6520^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-6520^4,DevicePath:,},},Config:nil,},} Dec 1 23:27:10.096: INFO: Logging kubelet events for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:27:10.276: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:27:10.461: INFO: webserver-deployment-847dcfb7fb-7mvn2 started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container httpd ready: false, restart count 0 Dec 1 23:27:10.461: INFO: csi-mockplugin-0 started at 2022-12-01 23:21:48 +0000 UTC (0+3 container statuses recorded) Dec 1 23:27:10.461: INFO: Container csi-provisioner ready: false, restart count 0 Dec 1 23:27:10.461: INFO: Container driver-registrar ready: false, restart count 0 Dec 1 23:27:10.461: INFO: Container mock ready: false, restart count 0 Dec 1 23:27:10.461: INFO: netserver-2 started at 2022-12-01 23:21:41 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container webserver ready: true, restart count 0 Dec 1 23:27:10.461: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-8gc5h started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:10.461: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-5z2kc started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:10.461: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-nbwwt started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:27:10.461: INFO: csi-mockplugin-attacher-0 started at 2022-12-01 23:21:48 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:27:10.461: INFO: netserver-2 started at 2022-12-01 23:22:07 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container webserver ready: false, restart count 0 Dec 1 23:27:10.461: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-wzpt2 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:10.461: INFO: csi-mockplugin-0 started at 2022-12-01 23:21:21 +0000 UTC (0+3 container statuses recorded) Dec 1 23:27:10.461: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:27:10.461: INFO: Container driver-registrar ready: true, restart count 0 Dec 1 23:27:10.461: INFO: Container mock ready: true, restart count 0 Dec 1 23:27:10.461: INFO: ebs-csi-node-5zdr7 started at 2022-12-01 23:17:15 +0000 UTC (0+3 container statuses recorded) Dec 1 23:27:10.461: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:27:10.461: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:27:10.461: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:27:10.461: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-vdxsm started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:27:10.461: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-ddtfs started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:27:10.461: INFO: cilium-nwg58 started at 2022-12-01 23:17:15 +0000 UTC (1+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:27:10.461: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:27:10.461: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-8l4bh started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:10.461: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-p8zf2 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:10.461: INFO: csi-mockplugin-attacher-0 started at 2022-12-01 23:21:21 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:27:10.461: INFO: coredns-5fcc7b6498-x9kjd started at 2022-12-01 23:17:56 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container coredns ready: true, restart count 0 Dec 1 23:27:10.461: INFO: netserver-2 started at 2022-12-01 23:22:25 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container webserver ready: false, restart count 0 Dec 1 23:27:10.461: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-nhwvz started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:10.461: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-fmm4p started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:10.461: INFO: webserver-deployment-847dcfb7fb-ddpw2 started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:10.461: INFO: Container httpd ready: false, restart count 0 Dec 1 23:27:11.087: INFO: Latency metrics for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:27:11.087: INFO: Logging node info for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:27:11.263: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-60-229.ap-northeast-2.compute.internal 63277852-d67d-402c-b963-ae3a2995d791 5538 0 2022-12-01 23:17:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-60-229.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-60-229.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-9901":"ip-172-20-60-229.ap-northeast-2.compute.internal","ebs.csi.aws.com":"i-0441b4e9d8ac12445"} io.cilium.network.ipv4-cilium-host:172.20.50.202 io.cilium.network.ipv4-health-ip:172.20.42.191 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:45 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:21:19 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2022-12-01 23:22:08 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-0441b4e9d8ac12445,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:45 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:22:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:22:41 +0000 UTC,LastTransitionTime:2022-12-01 23:17:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.229,},NodeAddress{Type:ExternalIP,Address:43.201.70.39,},NodeAddress{Type:Hostname,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-43-201-70-39.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec24f7651bea0c9073fca065a65b3820,SystemUUID:ec24f765-1bea-0c90-73fc-a065a65b3820,BootID:5386d7df-0b64-4c56-85ff-3f8a9a84c900,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:27:11.264: INFO: Logging kubelet events for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:27:11.442: INFO: Logging pods the kubelet thinks is on node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:27:11.626: INFO: execpod-affinitytn96q started at 2022-12-01 23:22:01 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container agnhost-container ready: false, restart count 0 Dec 1 23:27:11.627: INFO: csi-mockplugin-attacher-0 started at 2022-12-01 23:21:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container csi-attacher ready: false, restart count 0 Dec 1 23:27:11.627: INFO: webserver-deployment-847dcfb7fb-pjghl started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container httpd ready: false, restart count 0 Dec 1 23:27:11.627: INFO: busybox-privileged-true-6c52d94f-d5fa-47d4-8b09-d177a9cfb670 started at 2022-12-01 23:26:12 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container busybox-privileged-true-6c52d94f-d5fa-47d4-8b09-d177a9cfb670 ready: false, restart count 0 Dec 1 23:27:11.627: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-g98kz started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:27:11.627: INFO: sample-webhook-deployment-5d5bc8ffd5-4frfk started at 2022-12-01 23:22:06 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container sample-webhook ready: true, restart count 0 Dec 1 23:27:11.627: INFO: inline-volume-tester-8gmln started at 2022-12-01 23:21:58 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container csi-volume-tester ready: false, restart count 0 Dec 1 23:27:11.627: INFO: hostexec-ip-172-20-60-229.ap-northeast-2.compute.internal-5m52t started at 2022-12-01 23:27:05 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:27:11.627: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-zvfn4 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:11.627: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-7vclf started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:11.627: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-2hlfg started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:11.627: INFO: csi-hostpathplugin-0 started at 2022-12-01 23:21:57 +0000 UTC (0+7 container statuses recorded) Dec 1 23:27:11.627: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:27:11.627: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:27:11.627: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:27:11.627: INFO: Container csi-snapshotter ready: true, restart count 0 Dec 1 23:27:11.627: INFO: Container hostpath ready: true, restart count 0 Dec 1 23:27:11.627: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:27:11.627: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:27:11.627: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-szrp5 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:11.627: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-fczc8 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: false, restart count 0 Dec 1 23:27:11.627: INFO: alpine-nnp-true-e2ab7c9b-593a-4110-af8d-d26b2a9ae321 started at 2022-12-01 23:22:00 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container alpine-nnp-true-e2ab7c9b-593a-4110-af8d-d26b2a9ae321 ready: false, restart count 0 Dec 1 23:27:11.627: INFO: to-be-attached-pod started at 2022-12-01 23:22:22 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container container1 ready: false, restart count 0 Dec 1 23:27:11.627: INFO: netserver-3 started at 2022-12-01 23:22:25 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container webserver ready: false, restart count 0 Dec 1 23:27:11.627: INFO: pod-adoption started at 2022-12-01 23:26:53 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container pod-adoption ready: false, restart count 0 Dec 1 23:27:11.627: INFO: ebs-csi-node-wqh44 started at 2022-12-01 23:17:12 +0000 UTC (0+3 container statuses recorded) Dec 1 23:27:11.627: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:27:11.627: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:27:11.627: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:27:11.627: INFO: test-container-pod started at 2022-12-01 23:22:05 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container webserver ready: false, restart count 0 Dec 1 23:27:11.627: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-nr4j5 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:11.627: INFO: webserver-deployment-847dcfb7fb-xgjww started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container httpd ready: false, restart count 0 Dec 1 23:27:11.627: INFO: webserver-deployment-847dcfb7fb-wkgvk started at 2022-12-01 23:22:28 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container httpd ready: false, restart count 0 Dec 1 23:27:11.627: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-c4nx7 started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:11.627: INFO: csi-mockplugin-resizer-0 started at 2022-12-01 23:21:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container csi-resizer ready: false, restart count 0 Dec 1 23:27:11.627: INFO: host-test-container-pod started at 2022-12-01 23:22:06 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:27:11.627: INFO: cilium-5dnwd started at 2022-12-01 23:17:12 +0000 UTC (1+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:27:11.627: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:27:11.627: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-4p7lw started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:11.627: INFO: cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609-gvcsf started at 2022-12-01 23:21:20 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container cleanup40-decca635-aac9-4d8a-bcce-e429bbe36609 ready: true, restart count 0 Dec 1 23:27:11.627: INFO: netserver-3 started at 2022-12-01 23:21:42 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container webserver ready: true, restart count 0 Dec 1 23:27:11.627: INFO: csi-mockplugin-0 started at 2022-12-01 23:21:28 +0000 UTC (0+3 container statuses recorded) Dec 1 23:27:11.627: INFO: Container csi-provisioner ready: false, restart count 0 Dec 1 23:27:11.627: INFO: Container driver-registrar ready: false, restart count 0 Dec 1 23:27:11.627: INFO: Container mock ready: false, restart count 0 Dec 1 23:27:11.627: INFO: netserver-3 started at 2022-12-01 23:22:07 +0000 UTC (0+1 container statuses recorded) Dec 1 23:27:11.627: INFO: Container webserver ready: false, restart count 0 Dec 1 23:27:12.317: INFO: Latency metrics for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:27:12.317: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "fsgroupchangepolicy-4071" for this suite.
Find pod-67758a14-58b2-42e1-97fe-7d875edc1629 mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(OnRootMismatch\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\svolume\scontents\sownership\schanged\sin\sfirst\spod\,\snew\spod\swith\sdifferent\sfsgroup\sapplied\sto\sthe\svolume\scontents$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Dec 2 00:02:53.194: Unexpected error: <*errors.errorString | 0xc002df3270>: { s: "pod \"pod-c4aba9a9-fa58-46f6-a83c-47d0b1c0a73e\" is not Running: timed out waiting for the condition", } pod "pod-c4aba9a9-fa58-46f6-a83c-47d0b1c0a73e" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250from junit_22.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Dec 1 23:57:51.055: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename fsgroupchangepolicy �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with different fsgroup applied to the volume contents /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Dec 1 23:57:51.956: INFO: Creating resource for dynamic PV Dec 1 23:57:51.956: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass fsgroupchangepolicy-2679-e2e-scjqcpd �[1mSTEP�[0m: creating a claim Dec 1 23:57:52.133: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-2679 with fsgroup 1000 Dec 2 00:02:53.194: FAIL: Unexpected error: <*errors.errorString | 0xc002df3270>: { s: "pod \"pod-c4aba9a9-fa58-46f6-a83c-47d0b1c0a73e\" is not Running: timed out waiting for the condition", } pod "pod-c4aba9a9-fa58-46f6-a83c-47d0b1c0a73e" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc004185080, 0xc0027230d8, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250 +0x1c5 k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:221 +0x219 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc00037b800) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc00037b800) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc00037b800, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 �[1mSTEP�[0m: Deleting pvc Dec 2 00:02:53.371: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comsnt77" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "fsgroupchangepolicy-2679". �[1mSTEP�[0m: Found 6 events. Dec 2 00:02:53.904: INFO: At 2022-12-01 23:57:52 +0000 UTC - event for ebs.csi.aws.comsnt77: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Dec 2 00:02:53.904: INFO: At 2022-12-01 23:57:52 +0000 UTC - event for ebs.csi.aws.comsnt77: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Dec 2 00:02:53.904: INFO: At 2022-12-01 23:57:52 +0000 UTC - event for ebs.csi.aws.comsnt77: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-2679/ebs.csi.aws.comsnt77" Dec 2 00:02:53.904: INFO: At 2022-12-01 23:58:02 +0000 UTC - event for ebs.csi.aws.comsnt77: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-2679-e2e-scjqcpd": rpc error: code = DeadlineExceeded desc = context deadline exceeded Dec 2 00:02:53.904: INFO: At 2022-12-01 23:58:39 +0000 UTC - event for ebs.csi.aws.comsnt77: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-2679-e2e-scjqcpd": rpc error: code = Internal desc = Could not create volume "pvc-9d8a8fc7-e9c0-4be1-9f45-372a0151d95f": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Dec 2 00:02:53.904: INFO: At 2022-12-02 00:02:53 +0000 UTC - event for pod-c4aba9a9-fa58-46f6-a83c-47d0b1c0a73e: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-2679/ebs.csi.aws.comsnt77" Dec 2 00:02:54.080: INFO: POD NODE PHASE GRACE CONDITIONS Dec 2 00:02:54.080: INFO: pod-c4aba9a9-fa58-46f6-a83c-47d0b1c0a73e Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2022-12-02 00:02:53 +0000 UTC SchedulerError running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-2679/ebs.csi.aws.comsnt77"}] Dec 2 00:02:54.080: INFO: Dec 2 00:02:54.258: INFO: Logging node info for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:02:54.435: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-156.ap-northeast-2.compute.internal e92fe5b6-ec1b-421a-992a-1182ff1e3522 48951 0 2022-12-01 23:15:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-156.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03fe1aa2fac33129f"} io.cilium.network.ipv4-cilium-host:172.20.40.211 io.cilium.network.ipv4-health-ip:172.20.46.196 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-12-01 23:15:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-12-01 23:15:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-12-01 23:15:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:16:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:21 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03fe1aa2fac33129f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:21 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:58:02 +0000 UTC,LastTransitionTime:2022-12-01 23:16:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.156,},NodeAddress{Type:ExternalIP,Address:15.165.75.247,},NodeAddress{Type:Hostname,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-165-75-247.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccbd8e0eb399c0115bc276d51c7ab,SystemUUID:ec2ccbd8-e0eb-399c-0115-bc276d51c7ab,BootID:4c45c6e5-af61-46fe-b037-b7cc919a53fc,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.2],SizeBytes:41958567,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.2],SizeBytes:41441104,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.2],SizeBytes:4954598,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:02:54.436: INFO: Logging kubelet events for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:02:54.615: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:02:54.797: INFO: cilium-pb6vg started at 2022-12-01 23:15:46 +0000 UTC (1+1 container statuses recorded) Dec 2 00:02:54.797: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:02:54.797: INFO: Container cilium-agent ready: true, restart count 1 Dec 2 00:02:54.797: INFO: ebs-csi-controller-764b64d679-5qsrv started at 2022-12-01 23:15:46 +0000 UTC (0+5 container statuses recorded) Dec 2 00:02:54.797: INFO: Container csi-attacher ready: true, restart count 0 Dec 2 00:02:54.797: INFO: Container csi-provisioner ready: true, restart count 0 Dec 2 00:02:54.797: INFO: Container csi-resizer ready: true, restart count 0 Dec 2 00:02:54.797: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:02:54.797: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:02:54.797: INFO: kube-controller-manager-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:54.797: INFO: Container kube-controller-manager ready: true, restart count 2 Dec 2 00:02:54.797: INFO: etcd-manager-events-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:54.797: INFO: Container etcd-manager ready: true, restart count 0 Dec 2 00:02:54.797: INFO: etcd-manager-main-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:54.797: INFO: Container etcd-manager ready: true, restart count 0 Dec 2 00:02:54.797: INFO: kops-controller-g4pqg started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:54.797: INFO: Container kops-controller ready: true, restart count 0 Dec 2 00:02:54.797: INFO: cilium-operator-587dd66f87-pgs67 started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:54.797: INFO: Container cilium-operator ready: true, restart count 0 Dec 2 00:02:54.797: INFO: kube-scheduler-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:54.797: INFO: Container kube-scheduler ready: true, restart count 0 Dec 2 00:02:54.797: INFO: kube-apiserver-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+2 container statuses recorded) Dec 2 00:02:54.797: INFO: Container healthcheck ready: true, restart count 0 Dec 2 00:02:54.797: INFO: Container kube-apiserver ready: true, restart count 1 Dec 2 00:02:54.797: INFO: ebs-csi-node-m7cgg started at 2022-12-01 23:15:46 +0000 UTC (0+3 container statuses recorded) Dec 2 00:02:54.797: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:02:54.797: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:02:54.797: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:02:54.797: INFO: dns-controller-5fb778d9d5-qrfkv started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:54.797: INFO: Container dns-controller ready: true, restart count 0 Dec 2 00:02:55.390: INFO: Latency metrics for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 2 00:02:55.391: INFO: Logging node info for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:02:55.568: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-46-84.ap-northeast-2.compute.internal e408f2b8-c522-4e88-973a-78a2b7a2298f 50362 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-46-84.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-3465 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-46-84.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03a01a80f538d5d6b"} io.cilium.network.ipv4-cilium-host:172.20.38.72 io.cilium.network.ipv4-health-ip:172.20.48.223 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:40 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:40:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kubelet Update v1 2022-12-01 23:57:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03a01a80f538d5d6b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:40 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-02 00:02:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-02 00:02:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-02 00:02:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-02 00:02:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.46.84,},NodeAddress{Type:ExternalIP,Address:54.180.162.145,},NodeAddress{Type:Hostname,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-162-145.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec285e4d0d3397ed7e702d489e42654b,SystemUUID:ec285e4d-0d33-97ed-7e70-2d489e42654b,BootID:d6d1d15f-8121-4e40-b71c-e5b51419dc1b,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:02:55.568: INFO: Logging kubelet events for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:02:55.747: INFO: Logging pods the kubelet thinks is on node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:02:55.926: INFO: cilium-7h664 started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 2 00:02:55.926: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:02:55.926: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:02:55.926: INFO: ebs-csi-node-lfxzl started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 2 00:02:55.926: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:02:55.926: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:02:55.926: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:02:56.513: INFO: Latency metrics for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 2 00:02:56.513: INFO: Logging node info for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:02:56.690: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-54-150.ap-northeast-2.compute.internal 2a8a1d71-d00b-4259-82bd-841f96508593 50495 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-54-150.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-54-150.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02d689db76221fe72"} io.cilium.network.ipv4-cilium-host:172.20.43.0 io.cilium.network.ipv4-health-ip:172.20.58.253 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-12-01 23:17:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:38 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:56:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02d689db76221fe72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:38 +0000 UTC,LastTransitionTime:2022-12-01 23:17:38 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-02 00:02:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-02 00:02:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-02 00:02:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-02 00:02:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:30 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.150,},NodeAddress{Type:ExternalIP,Address:54.180.133.96,},NodeAddress{Type:Hostname,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-133-96.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec20870a7ff0c6740650f49249abc430,SystemUUID:ec20870a-7ff0-c674-0650-f49249abc430,BootID:803a40bf-5805-4ae9-b933-fde612d75f1d,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 k8s.gcr.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:e7fddbaac4c3451da2365ab90bad149d32f11409738034e41e0f460927f7c276 k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.4],SizeBytes:24757245,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:02:56.691: INFO: Logging kubelet events for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:02:56.871: INFO: Logging pods the kubelet thinks is on node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:02:57.051: INFO: cilium-2bxvd started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 2 00:02:57.051: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:02:57.051: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:02:57.051: INFO: coredns-5fcc7b6498-vcsml started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:57.051: INFO: Container coredns ready: true, restart count 0 Dec 2 00:02:57.051: INFO: coredns-autoscaler-6658b4bf85-92nhw started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:57.051: INFO: Container autoscaler ready: true, restart count 0 Dec 2 00:02:57.051: INFO: ebs-csi-node-n5tcr started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 2 00:02:57.051: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:02:57.051: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:02:57.051: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:02:57.637: INFO: Latency metrics for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 2 00:02:57.637: INFO: Logging node info for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:02:57.814: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-57.ap-northeast-2.compute.internal 2ea0d3e0-dcd9-4562-ada8-ee5e6b866480 50550 0 2022-12-01 23:17:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-57.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-55-57.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02200981456309a72"} io.cilium.network.ipv4-cilium-host:172.20.51.99 io.cilium.network.ipv4-health-ip:172.20.36.120 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:44 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:54:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02200981456309a72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:44 +0000 UTC,LastTransitionTime:2022-12-01 23:17:44 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-02 00:02:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-02 00:02:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-02 00:02:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-02 00:02:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.57,},NodeAddress{Type:ExternalIP,Address:3.36.130.97,},NodeAddress{Type:Hostname,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-36-130-97.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bf2fd9fd22cd1aac5da0aa67f7a08,SystemUUID:ec2bf2fd-9fd2-2cd1-aac5-da0aa67f7a08,BootID:c4c37f48-1b62-417d-82bf-ea460350f712,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-936^47b47419-71d1-11ed-be23-02bdc8e24a07],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:02:57.815: INFO: Logging kubelet events for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:02:57.994: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:02:58.174: INFO: ebs-csi-node-5zdr7 started at 2022-12-01 23:17:15 +0000 UTC (0+3 container statuses recorded) Dec 2 00:02:58.174: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:02:58.174: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:02:58.174: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:02:58.174: INFO: cilium-nwg58 started at 2022-12-01 23:17:15 +0000 UTC (1+1 container statuses recorded) Dec 2 00:02:58.174: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:02:58.174: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:02:58.174: INFO: coredns-5fcc7b6498-x9kjd started at 2022-12-01 23:17:56 +0000 UTC (0+1 container statuses recorded) Dec 2 00:02:58.174: INFO: Container coredns ready: true, restart count 0 Dec 2 00:02:58.818: INFO: Latency metrics for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 2 00:02:58.818: INFO: Logging node info for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:02:58.995: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-60-229.ap-northeast-2.compute.internal 63277852-d67d-402c-b963-ae3a2995d791 49657 0 2022-12-01 23:17:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-60-229.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-60-229.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0441b4e9d8ac12445"} io.cilium.network.ipv4-cilium-host:172.20.50.202 io.cilium.network.ipv4-health-ip:172.20.42.191 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:45 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:59:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-0441b4e9d8ac12445,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:45 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:59:13 +0000 UTC,LastTransitionTime:2022-12-01 23:17:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.229,},NodeAddress{Type:ExternalIP,Address:43.201.70.39,},NodeAddress{Type:Hostname,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-43-201-70-39.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec24f7651bea0c9073fca065a65b3820,SystemUUID:ec24f765-1bea-0c90-73fc-a065a65b3820,BootID:5386d7df-0b64-4c56-85ff-3f8a9a84c900,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 2 00:02:58.997: INFO: Logging kubelet events for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:02:59.176: INFO: Logging pods the kubelet thinks is on node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:02:59.355: INFO: ebs-csi-node-wqh44 started at 2022-12-01 23:17:12 +0000 UTC (0+3 container statuses recorded) Dec 2 00:02:59.355: INFO: Container ebs-plugin ready: true, restart count 0 Dec 2 00:02:59.355: INFO: Container liveness-probe ready: true, restart count 0 Dec 2 00:02:59.355: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 2 00:02:59.355: INFO: cilium-5dnwd started at 2022-12-01 23:17:12 +0000 UTC (1+1 container statuses recorded) Dec 2 00:02:59.355: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 2 00:02:59.355: INFO: Container cilium-agent ready: true, restart count 0 Dec 2 00:02:59.943: INFO: Latency metrics for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 2 00:02:59.943: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "fsgroupchangepolicy-2679" for this suite.
Find pod-c4aba9a9-fa58-46f6-a83c-47d0b1c0a73e mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(OnRootMismatch\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\svolume\scontents\sownership\schanged\sin\sfirst\spod\,\snew\spod\swith\ssame\sfsgroup\sskips\sownership\schanges\sto\sthe\svolume\scontents$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Dec 1 23:56:17.446: Unexpected error: <*errors.errorString | 0xc000ab9ad0>: { s: "pod \"pod-ebc59eaf-1351-48bb-89e6-811812cae1d6\" is not Running: timed out waiting for the condition", } pod "pod-ebc59eaf-1351-48bb-89e6-811812cae1d6" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250from junit_11.xml
{"msg":"PASSED [sig-node] Security Context when creating containers with AllowPrivilegeEscalation should allow privilege escalation when not explicitly set and uid != 0 [LinuxOnly] [NodeConformance]","total":-1,"completed":28,"skipped":216,"failed":2,"failures":["[sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support file as subpath [LinuxOnly]","External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand should resize volume when PVC is edited while pod is using it"]} [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Dec 1 23:51:15.321: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename fsgroupchangepolicy �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup skips ownership changes to the volume contents /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Dec 1 23:51:16.203: INFO: Creating resource for dynamic PV Dec 1 23:51:16.204: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass fsgroupchangepolicy-7104-e2e-sczbq7r �[1mSTEP�[0m: creating a claim Dec 1 23:51:16.381: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-7104 with fsgroup 1000 Dec 1 23:56:17.446: FAIL: Unexpected error: <*errors.errorString | 0xc000ab9ad0>: { s: "pod \"pod-ebc59eaf-1351-48bb-89e6-811812cae1d6\" is not Running: timed out waiting for the condition", } pod "pod-ebc59eaf-1351-48bb-89e6-811812cae1d6" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc0041306e0, 0xc002a130d8, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250 +0x1c5 k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:221 +0x219 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000726000) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000726000) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000726000, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 �[1mSTEP�[0m: Deleting pvc Dec 1 23:56:17.623: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comr4vvx" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "fsgroupchangepolicy-7104". �[1mSTEP�[0m: Found 6 events. Dec 1 23:56:18.156: INFO: At 2022-12-01 23:51:16 +0000 UTC - event for ebs.csi.aws.comr4vvx: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Dec 1 23:56:18.156: INFO: At 2022-12-01 23:51:16 +0000 UTC - event for ebs.csi.aws.comr4vvx: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Dec 1 23:56:18.156: INFO: At 2022-12-01 23:51:16 +0000 UTC - event for ebs.csi.aws.comr4vvx: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-7104/ebs.csi.aws.comr4vvx" Dec 1 23:56:18.156: INFO: At 2022-12-01 23:51:26 +0000 UTC - event for ebs.csi.aws.comr4vvx: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-7104-e2e-sczbq7r": rpc error: code = DeadlineExceeded desc = context deadline exceeded Dec 1 23:56:18.156: INFO: At 2022-12-01 23:51:37 +0000 UTC - event for ebs.csi.aws.comr4vvx: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-7104-e2e-sczbq7r": rpc error: code = Internal desc = Could not create volume "pvc-3b82e914-7661-4efc-aeea-b220e61c3d6e": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Dec 1 23:56:18.156: INFO: At 2022-12-01 23:56:17 +0000 UTC - event for pod-ebc59eaf-1351-48bb-89e6-811812cae1d6: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-7104/ebs.csi.aws.comr4vvx" Dec 1 23:56:18.332: INFO: POD NODE PHASE GRACE CONDITIONS Dec 1 23:56:18.332: INFO: pod-ebc59eaf-1351-48bb-89e6-811812cae1d6 Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2022-12-01 23:56:17 +0000 UTC SchedulerError running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-7104/ebs.csi.aws.comr4vvx"}] Dec 1 23:56:18.332: INFO: Dec 1 23:56:18.512: INFO: Logging node info for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:56:18.688: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-156.ap-northeast-2.compute.internal e92fe5b6-ec1b-421a-992a-1182ff1e3522 40174 0 2022-12-01 23:15:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-156.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03fe1aa2fac33129f"} io.cilium.network.ipv4-cilium-host:172.20.40.211 io.cilium.network.ipv4-health-ip:172.20.46.196 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-12-01 23:15:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-12-01 23:15:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-12-01 23:15:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:16:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:21 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03fe1aa2fac33129f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:21 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:53:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:53:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:53:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:53:02 +0000 UTC,LastTransitionTime:2022-12-01 23:16:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.156,},NodeAddress{Type:ExternalIP,Address:15.165.75.247,},NodeAddress{Type:Hostname,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-165-75-247.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccbd8e0eb399c0115bc276d51c7ab,SystemUUID:ec2ccbd8-e0eb-399c-0115-bc276d51c7ab,BootID:4c45c6e5-af61-46fe-b037-b7cc919a53fc,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.2],SizeBytes:41958567,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.2],SizeBytes:41441104,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.2],SizeBytes:4954598,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:56:18.689: INFO: Logging kubelet events for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:56:18.867: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:56:19.053: INFO: kube-scheduler-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:19.053: INFO: Container kube-scheduler ready: true, restart count 0 Dec 1 23:56:19.053: INFO: kube-apiserver-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+2 container statuses recorded) Dec 1 23:56:19.053: INFO: Container healthcheck ready: true, restart count 0 Dec 1 23:56:19.053: INFO: Container kube-apiserver ready: true, restart count 1 Dec 1 23:56:19.053: INFO: ebs-csi-node-m7cgg started at 2022-12-01 23:15:46 +0000 UTC (0+3 container statuses recorded) Dec 1 23:56:19.053: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:56:19.053: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:56:19.053: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:56:19.053: INFO: dns-controller-5fb778d9d5-qrfkv started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:19.053: INFO: Container dns-controller ready: true, restart count 0 Dec 1 23:56:19.053: INFO: cilium-operator-587dd66f87-pgs67 started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:19.053: INFO: Container cilium-operator ready: true, restart count 0 Dec 1 23:56:19.053: INFO: ebs-csi-controller-764b64d679-5qsrv started at 2022-12-01 23:15:46 +0000 UTC (0+5 container statuses recorded) Dec 1 23:56:19.053: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:56:19.053: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:56:19.053: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:56:19.053: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:56:19.053: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:56:19.053: INFO: kube-controller-manager-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:19.053: INFO: Container kube-controller-manager ready: true, restart count 2 Dec 1 23:56:19.053: INFO: etcd-manager-events-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:19.053: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:56:19.053: INFO: etcd-manager-main-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:19.053: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:56:19.053: INFO: kops-controller-g4pqg started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:19.053: INFO: Container kops-controller ready: true, restart count 0 Dec 1 23:56:19.053: INFO: cilium-pb6vg started at 2022-12-01 23:15:46 +0000 UTC (1+1 container statuses recorded) Dec 1 23:56:19.053: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:56:19.053: INFO: Container cilium-agent ready: true, restart count 1 Dec 1 23:56:19.640: INFO: Latency metrics for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:56:19.640: INFO: Logging node info for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:56:19.817: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-46-84.ap-northeast-2.compute.internal e408f2b8-c522-4e88-973a-78a2b7a2298f 43703 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-46-84.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-3465 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-46-84.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03a01a80f538d5d6b"} io.cilium.network.ipv4-cilium-host:172.20.38.72 io.cilium.network.ipv4-health-ip:172.20.48.223 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:40 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:40:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kubelet Update v1 2022-12-01 23:54:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03a01a80f538d5d6b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:40 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:54:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:54:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:54:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:54:42 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.46.84,},NodeAddress{Type:ExternalIP,Address:54.180.162.145,},NodeAddress{Type:Hostname,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-162-145.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec285e4d0d3397ed7e702d489e42654b,SystemUUID:ec285e4d-0d33-97ed-7e70-2d489e42654b,BootID:d6d1d15f-8121-4e40-b71c-e5b51419dc1b,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:56:19.817: INFO: Logging kubelet events for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:56:19.996: INFO: Logging pods the kubelet thinks is on node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:56:20.178: INFO: sample-webhook-deployment-5d5bc8ffd5-w8s4q started at 2022-12-01 23:56:14 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:20.178: INFO: Container sample-webhook ready: true, restart count 0 Dec 1 23:56:20.178: INFO: ebs-csi-node-lfxzl started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:56:20.178: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:56:20.178: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:56:20.178: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:56:20.178: INFO: pod-ready started at 2022-12-01 23:55:25 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:20.178: INFO: Container pod-readiness-gate ready: true, restart count 0 Dec 1 23:56:20.178: INFO: netserver-0 started at 2022-12-01 23:56:13 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:20.178: INFO: Container webserver ready: false, restart count 0 Dec 1 23:56:20.178: INFO: send-events-7de62c69-21a9-48c2-8fc2-05f9c8bf7f4e started at 2022-12-01 23:55:50 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:20.178: INFO: Container p ready: true, restart count 0 Dec 1 23:56:20.178: INFO: cilium-7h664 started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:56:20.178: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:56:20.178: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:56:20.178: INFO: netserver-0 started at 2022-12-01 23:56:17 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:20.178: INFO: Container webserver ready: false, restart count 0 Dec 1 23:56:20.178: INFO: hostexec-ip-172-20-46-84.ap-northeast-2.compute.internal-rq2vx started at 2022-12-01 23:56:11 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:20.178: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:56:20.178: INFO: run-test started at 2022-12-01 23:56:17 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:20.178: INFO: Container run-test ready: true, restart count 0 Dec 1 23:56:20.178: INFO: pod-6facff0b-d1bd-43d8-be7d-5b52dacc6861 started at 2022-12-01 23:56:18 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:20.178: INFO: Container write-pod ready: false, restart count 0 Dec 1 23:56:20.178: INFO: netserver-0 started at 2022-12-01 23:56:05 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:20.178: INFO: Container webserver ready: false, restart count 0 Dec 1 23:56:20.809: INFO: Latency metrics for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:56:20.809: INFO: Logging node info for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:56:20.985: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-54-150.ap-northeast-2.compute.internal 2a8a1d71-d00b-4259-82bd-841f96508593 45353 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-54-150.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-54-150.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-2567":"ip-172-20-54-150.ap-northeast-2.compute.internal","ebs.csi.aws.com":"i-02d689db76221fe72"} io.cilium.network.ipv4-cilium-host:172.20.43.0 io.cilium.network.ipv4-health-ip:172.20.58.253 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-12-01 23:17:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:38 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-12-01 23:55:34 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-12-01 23:55:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02d689db76221fe72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:38 +0000 UTC,LastTransitionTime:2022-12-01 23:17:38 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:56:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:56:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:56:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:56:12 +0000 UTC,LastTransitionTime:2022-12-01 23:17:30 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.150,},NodeAddress{Type:ExternalIP,Address:54.180.133.96,},NodeAddress{Type:Hostname,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-133-96.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec20870a7ff0c6740650f49249abc430,SystemUUID:ec20870a-7ff0-c674-0650-f49249abc430,BootID:803a40bf-5805-4ae9-b933-fde612d75f1d,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 k8s.gcr.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:e7fddbaac4c3451da2365ab90bad149d32f11409738034e41e0f460927f7c276 k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.4],SizeBytes:24757245,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-provisioning-2567^b85bcd3c-71d3-11ed-83ff-3a0c13bf9040],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-2567^b85bcd3c-71d3-11ed-83ff-3a0c13bf9040,DevicePath:,},},Config:nil,},} Dec 1 23:56:20.986: INFO: Logging kubelet events for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:56:21.167: INFO: Logging pods the kubelet thinks is on node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:56:21.350: INFO: default started at 2022-12-01 23:55:18 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:21.350: INFO: Container cntr ready: true, restart count 0 Dec 1 23:56:21.350: INFO: ebs-csi-node-n5tcr started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:56:21.350: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:56:21.350: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:56:21.350: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:56:21.350: INFO: concurrent-27832315-2sbt6 started at 2022-12-01 23:55:00 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:21.350: INFO: Container c ready: true, restart count 0 Dec 1 23:56:21.350: INFO: slave started at 2022-12-01 23:55:13 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:21.350: INFO: Container cntr ready: true, restart count 0 Dec 1 23:56:21.350: INFO: private started at 2022-12-01 23:55:15 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:21.350: INFO: Container cntr ready: true, restart count 0 Dec 1 23:56:21.350: INFO: cilium-2bxvd started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:56:21.350: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:56:21.350: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:56:21.350: INFO: coredns-5fcc7b6498-vcsml started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:21.350: INFO: Container coredns ready: true, restart count 0 Dec 1 23:56:21.350: INFO: hostexec-ip-172-20-54-150.ap-northeast-2.compute.internal-4r8vm started at 2022-12-01 23:55:48 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:21.350: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:56:21.350: INFO: netserver-1 started at 2022-12-01 23:56:13 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:21.350: INFO: Container webserver ready: false, restart count 0 Dec 1 23:56:21.350: INFO: netserver-1 started at 2022-12-01 23:56:17 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:21.350: INFO: Container webserver ready: false, restart count 0 Dec 1 23:56:21.350: INFO: netserver-1 started at 2022-12-01 23:56:05 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:21.350: INFO: Container webserver ready: false, restart count 0 Dec 1 23:56:21.350: INFO: coredns-autoscaler-6658b4bf85-92nhw started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:21.350: INFO: Container autoscaler ready: true, restart count 0 Dec 1 23:56:21.350: INFO: master started at 2022-12-01 23:55:08 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:21.350: INFO: Container cntr ready: true, restart count 0 Dec 1 23:56:21.350: INFO: csi-hostpathplugin-0 started at 2022-12-01 23:55:38 +0000 UTC (0+7 container statuses recorded) Dec 1 23:56:21.350: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:56:21.350: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:56:21.350: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:56:21.350: INFO: Container csi-snapshotter ready: true, restart count 0 Dec 1 23:56:21.350: INFO: Container hostpath ready: true, restart count 0 Dec 1 23:56:21.350: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:56:21.350: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:56:21.971: INFO: Latency metrics for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:56:21.971: INFO: Logging node info for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:56:22.148: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-57.ap-northeast-2.compute.internal 2ea0d3e0-dcd9-4562-ada8-ee5e6b866480 44067 0 2022-12-01 23:17:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-57.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-55-57.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02200981456309a72"} io.cilium.network.ipv4-cilium-host:172.20.51.99 io.cilium.network.ipv4-health-ip:172.20.36.120 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:44 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:54:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02200981456309a72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:44 +0000 UTC,LastTransitionTime:2022-12-01 23:17:44 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:55:16 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:55:16 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:55:16 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:55:16 +0000 UTC,LastTransitionTime:2022-12-01 23:17:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.57,},NodeAddress{Type:ExternalIP,Address:3.36.130.97,},NodeAddress{Type:Hostname,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-57.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-36-130-97.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2bf2fd9fd22cd1aac5da0aa67f7a08,SystemUUID:ec2bf2fd-9fd2-2cd1-aac5-da0aa67f7a08,BootID:c4c37f48-1b62-417d-82bf-ea460350f712,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-936^47b47419-71d1-11ed-be23-02bdc8e24a07],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:56:22.149: INFO: Logging kubelet events for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:56:22.328: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:56:22.508: INFO: ebs-csi-node-5zdr7 started at 2022-12-01 23:17:15 +0000 UTC (0+3 container statuses recorded) Dec 1 23:56:22.508: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:56:22.508: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:56:22.508: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:56:22.508: INFO: httpd started at 2022-12-01 23:56:05 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:22.508: INFO: Container httpd ready: true, restart count 0 Dec 1 23:56:22.508: INFO: netserver-2 started at 2022-12-01 23:56:13 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:22.508: INFO: Container webserver ready: false, restart count 0 Dec 1 23:56:22.508: INFO: cilium-nwg58 started at 2022-12-01 23:17:15 +0000 UTC (1+1 container statuses recorded) Dec 1 23:56:22.508: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:56:22.508: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:56:22.508: INFO: coredns-5fcc7b6498-x9kjd started at 2022-12-01 23:17:56 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:22.509: INFO: Container coredns ready: true, restart count 0 Dec 1 23:56:22.509: INFO: liveness-fec49b40-d339-47c4-9e6d-675f321d9503 started at 2022-12-01 23:52:42 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:22.509: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:56:22.509: INFO: netserver-2 started at 2022-12-01 23:56:05 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:22.509: INFO: Container webserver ready: false, restart count 0 Dec 1 23:56:22.509: INFO: test-pod started at 2022-12-01 23:56:21 +0000 UTC (0+3 container statuses recorded) Dec 1 23:56:22.509: INFO: Container busybox-1 ready: false, restart count 0 Dec 1 23:56:22.509: INFO: Container busybox-2 ready: false, restart count 0 Dec 1 23:56:22.509: INFO: Container busybox-3 ready: false, restart count 0 Dec 1 23:56:22.509: INFO: netserver-2 started at 2022-12-01 23:56:17 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:22.509: INFO: Container webserver ready: false, restart count 0 Dec 1 23:56:23.206: INFO: Latency metrics for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:56:23.206: INFO: Logging node info for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:56:23.383: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-60-229.ap-northeast-2.compute.internal 63277852-d67d-402c-b963-ae3a2995d791 45675 0 2022-12-01 23:17:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-60-229.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-60-229.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0441b4e9d8ac12445"} io.cilium.network.ipv4-cilium-host:172.20.50.202 io.cilium.network.ipv4-health-ip:172.20.42.191 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:45 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:51:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-0441b4e9d8ac12445,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051689472 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946831872 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:45 +0000 UTC,LastTransitionTime:2022-12-01 23:17:45 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:56:22 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:56:22 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:56:22 +0000 UTC,LastTransitionTime:2022-12-01 23:17:11 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:56:22 +0000 UTC,LastTransitionTime:2022-12-01 23:17:41 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.60.229,},NodeAddress{Type:ExternalIP,Address:43.201.70.39,},NodeAddress{Type:Hostname,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-60-229.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-43-201-70-39.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec24f7651bea0c9073fca065a65b3820,SystemUUID:ec24f765-1bea-0c90-73fc-a065a65b3820,BootID:5386d7df-0b64-4c56-85ff-3f8a9a84c900,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:56:23.383: INFO: Logging kubelet events for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:56:23.562: INFO: Logging pods the kubelet thinks is on node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:56:23.742: INFO: netserver-3 started at 2022-12-01 23:56:17 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:23.743: INFO: Container webserver ready: false, restart count 0 Dec 1 23:56:23.743: INFO: hostexec-ip-172-20-60-229.ap-northeast-2.compute.internal-rpv58 started at 2022-12-01 23:56:21 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:23.743: INFO: Container agnhost-container ready: true, restart count 0 Dec 1 23:56:23.743: INFO: ebs-csi-node-wqh44 started at 2022-12-01 23:17:12 +0000 UTC (0+3 container statuses recorded) Dec 1 23:56:23.743: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:56:23.743: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:56:23.743: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:56:23.743: INFO: concurrent-27832316-jxpxb started at 2022-12-01 23:56:00 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:23.743: INFO: Container c ready: true, restart count 0 Dec 1 23:56:23.743: INFO: cilium-5dnwd started at 2022-12-01 23:17:12 +0000 UTC (1+1 container statuses recorded) Dec 1 23:56:23.743: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:56:23.743: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:56:23.743: INFO: netserver-3 started at 2022-12-01 23:56:13 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:23.743: INFO: Container webserver ready: false, restart count 0 Dec 1 23:56:23.743: INFO: netserver-3 started at 2022-12-01 23:56:06 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:23.743: INFO: Container webserver ready: false, restart count 0 Dec 1 23:56:23.743: INFO: test-deployment-6c9dc9cdf6-md797 started at 2022-12-01 23:56:23 +0000 UTC (0+1 container statuses recorded) Dec 1 23:56:23.743: INFO: Container test-deployment ready: false, restart count 0 Dec 1 23:56:24.392: INFO: Latency metrics for node ip-172-20-60-229.ap-northeast-2.compute.internal Dec 1 23:56:24.392: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "fsgroupchangepolicy-7104" for this suite.
Find pod-ebc59eaf-1351-48bb-89e6-811812cae1d6 mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sprovisioning\sshould\sprovision\sstorage\swith\smount\soptions$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:180 Dec 1 23:48:40.470: Unexpected error: <*errors.errorString | 0xc005446eb0>: { s: "pod \"pod-dc400206-f4d6-4ddc-b8d0-029adc0aae2b\" is not Running: timed out waiting for the condition", } pod "pod-dc400206-f4d6-4ddc-b8d0-029adc0aae2b" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:418
[BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Dec 1 23:43:37.995: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename provisioning �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should provision storage with mount options /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:180 Dec 1 23:43:38.879: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Dec 1 23:43:38.879: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Dec 1 23:43:38.879: INFO: In creating storage class object and pvc objects for driver - sc: &StorageClass{ObjectMeta:{provisioning-3959-e2e-sch979q 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Provisioner:ebs.csi.aws.com,Parameters:map[string]string{encrypted: true,type: gp3,},ReclaimPolicy:nil,MountOptions:[],AllowVolumeExpansion:*true,VolumeBindingMode:*WaitForFirstConsumer,AllowedTopologies:[]TopologySelectorTerm{},}, pvc: &PersistentVolumeClaim{ObjectMeta:{ pvc- provisioning-3959 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*provisioning-3959-e2e-sch979q,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},}, src-pvc: &PersistentVolumeClaim{ObjectMeta:{ pvc- provisioning-3959 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*provisioning-3959-e2e-sch979q,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} �[1mSTEP�[0m: Creating a StorageClass �[1mSTEP�[0m: creating claim=&PersistentVolumeClaim{ObjectMeta:{ pvc- provisioning-3959 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*provisioning-3959-e2e-sch979q,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} �[1mSTEP�[0m: creating a pod referring to the class=&StorageClass{ObjectMeta:{provisioning-3959-e2e-sch979q 2f67a3d9-951b-4482-b628-47a549966c6a 23405 0 2022-12-01 23:43:39 +0000 UTC <nil> <nil> map[] map[] [] [] [{e2e.test Update storage.k8s.io/v1 2022-12-01 23:43:39 +0000 UTC FieldsV1 {"f:allowVolumeExpansion":{},"f:mountOptions":{},"f:parameters":{".":{},"f:encrypted":{},"f:type":{}},"f:provisioner":{},"f:reclaimPolicy":{},"f:volumeBindingMode":{}} }]},Provisioner:ebs.csi.aws.com,Parameters:map[string]string{encrypted: true,type: gp3,},ReclaimPolicy:*Delete,MountOptions:[dirsync],AllowVolumeExpansion:*true,VolumeBindingMode:*WaitForFirstConsumer,AllowedTopologies:[]TopologySelectorTerm{},} claim=&PersistentVolumeClaim{ObjectMeta:{pvc-x57n6 pvc- provisioning-3959 66760cb8-6437-4f10-8a4a-33564aa2fb60 23418 0 2022-12-01 23:43:39 +0000 UTC <nil> <nil> map[] map[] [] [kubernetes.io/pvc-protection] [{e2e.test Update v1 2022-12-01 23:43:39 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} }]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*provisioning-3959-e2e-sch979q,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Pending,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} Dec 1 23:48:40.470: FAIL: Unexpected error: <*errors.errorString | 0xc005446eb0>: { s: "pod \"pod-dc400206-f4d6-4ddc-b8d0-029adc0aae2b\" is not Running: timed out waiting for the condition", } pod "pod-dc400206-f4d6-4ddc-b8d0-029adc0aae2b" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.StorageClassTest.TestDynamicProvisioning(0x7973fe8, 0xc0021669a0, 0xc003cb5b20, 0xc001e6a000, 0xc001e6a1c0, 0xc001b3aa80, 0x0, 0x0, 0x0, 0x0, ...) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:418 +0xe7d k8s.io/kubernetes/test/e2e/storage/testsuites.(*provisioningTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:198 +0x245 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000187980) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000187980) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000187980, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 Dec 1 23:48:40.471: INFO: deleting claim "provisioning-3959"/"pvc-x57n6" Dec 1 23:48:40.648: INFO: deleting storage class provisioning-3959-e2e-sch979q [AfterEach] [Testpattern: Dynamic PV (default fs)] provisioning /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "provisioning-3959". �[1mSTEP�[0m: Found 6 events. Dec 1 23:48:41.001: INFO: At 2022-12-01 23:43:39 +0000 UTC - event for pvc-x57n6: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Dec 1 23:48:41.001: INFO: At 2022-12-01 23:43:39 +0000 UTC - event for pvc-x57n6: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Dec 1 23:48:41.001: INFO: At 2022-12-01 23:43:39 +0000 UTC - event for pvc-x57n6: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } Provisioning: External provisioner is provisioning volume for claim "provisioning-3959/pvc-x57n6" Dec 1 23:48:41.001: INFO: At 2022-12-01 23:43:49 +0000 UTC - event for pvc-x57n6: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "provisioning-3959-e2e-sch979q": rpc error: code = Internal desc = Could not create volume "pvc-66760cb8-6437-4f10-8a4a-33564aa2fb60": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Dec 1 23:48:41.001: INFO: At 2022-12-01 23:44:00 +0000 UTC - event for pvc-x57n6: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-5qsrv_b10b2615-b259-434c-ae0c-d3fbbc30e639 } ProvisioningFailed: failed to provision volume with StorageClass "provisioning-3959-e2e-sch979q": rpc error: code = DeadlineExceeded desc = context deadline exceeded Dec 1 23:48:41.001: INFO: At 2022-12-01 23:48:40 +0000 UTC - event for pod-dc400206-f4d6-4ddc-b8d0-029adc0aae2b: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "provisioning-3959/pvc-x57n6" Dec 1 23:48:41.177: INFO: POD NODE PHASE GRACE CONDITIONS Dec 1 23:48:41.177: INFO: pod-dc400206-f4d6-4ddc-b8d0-029adc0aae2b Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2022-12-01 23:48:40 +0000 UTC SchedulerError running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "provisioning-3959/pvc-x57n6"}] Dec 1 23:48:41.177: INFO: Dec 1 23:48:41.354: INFO: Logging node info for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:48:41.530: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-38-156.ap-northeast-2.compute.internal e92fe5b6-ec1b-421a-992a-1182ff1e3522 32516 0 2022-12-01 23:15:11 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-38-156.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03fe1aa2fac33129f"} io.cilium.network.ipv4-cilium-host:172.20.40.211 io.cilium.network.ipv4-health-ip:172.20.46.196 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-12-01 23:15:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-12-01 23:15:37 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-12-01 23:15:47 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:16:01 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:21 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:17:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03fe1aa2fac33129f,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:21 +0000 UTC,LastTransitionTime:2022-12-01 23:17:21 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:48:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:48:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:48:02 +0000 UTC,LastTransitionTime:2022-12-01 23:15:06 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:48:02 +0000 UTC,LastTransitionTime:2022-12-01 23:16:01 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.156,},NodeAddress{Type:ExternalIP,Address:15.165.75.247,},NodeAddress{Type:Hostname,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-38-156.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-165-75-247.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccbd8e0eb399c0115bc276d51c7ab,SystemUUID:ec2ccbd8-e0eb-399c-0115-bc276d51c7ab,BootID:4c45c6e5-af61-46fe-b037-b7cc919a53fc,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.2],SizeBytes:41958567,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.2],SizeBytes:41441104,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.2],SizeBytes:4954598,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:48:41.531: INFO: Logging kubelet events for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:48:41.709: INFO: Logging pods the kubelet thinks is on node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:48:41.896: INFO: kube-scheduler-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:41.896: INFO: Container kube-scheduler ready: true, restart count 0 Dec 1 23:48:41.896: INFO: kube-apiserver-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+2 container statuses recorded) Dec 1 23:48:41.896: INFO: Container healthcheck ready: true, restart count 0 Dec 1 23:48:41.896: INFO: Container kube-apiserver ready: true, restart count 1 Dec 1 23:48:41.896: INFO: ebs-csi-node-m7cgg started at 2022-12-01 23:15:46 +0000 UTC (0+3 container statuses recorded) Dec 1 23:48:41.896: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:48:41.896: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:48:41.896: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:48:41.896: INFO: dns-controller-5fb778d9d5-qrfkv started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:41.896: INFO: Container dns-controller ready: true, restart count 0 Dec 1 23:48:41.896: INFO: cilium-operator-587dd66f87-pgs67 started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:41.896: INFO: Container cilium-operator ready: true, restart count 0 Dec 1 23:48:41.896: INFO: kube-controller-manager-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:41.896: INFO: Container kube-controller-manager ready: true, restart count 2 Dec 1 23:48:41.896: INFO: etcd-manager-events-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:41.896: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:48:41.896: INFO: etcd-manager-main-ip-172-20-38-156.ap-northeast-2.compute.internal started at 2022-12-01 23:14:29 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:41.896: INFO: Container etcd-manager ready: true, restart count 0 Dec 1 23:48:41.896: INFO: kops-controller-g4pqg started at 2022-12-01 23:15:46 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:41.896: INFO: Container kops-controller ready: true, restart count 0 Dec 1 23:48:41.896: INFO: cilium-pb6vg started at 2022-12-01 23:15:46 +0000 UTC (1+1 container statuses recorded) Dec 1 23:48:41.896: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:48:41.896: INFO: Container cilium-agent ready: true, restart count 1 Dec 1 23:48:41.896: INFO: ebs-csi-controller-764b64d679-5qsrv started at 2022-12-01 23:15:46 +0000 UTC (0+5 container statuses recorded) Dec 1 23:48:41.896: INFO: Container csi-attacher ready: true, restart count 0 Dec 1 23:48:41.896: INFO: Container csi-provisioner ready: true, restart count 0 Dec 1 23:48:41.896: INFO: Container csi-resizer ready: true, restart count 0 Dec 1 23:48:41.896: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:48:41.896: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:48:42.545: INFO: Latency metrics for node ip-172-20-38-156.ap-northeast-2.compute.internal Dec 1 23:48:42.545: INFO: Logging node info for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:48:42.721: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-46-84.ap-northeast-2.compute.internal e408f2b8-c522-4e88-973a-78a2b7a2298f 28204 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-46-84.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-3465 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-46-84.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-03a01a80f538d5d6b"} io.cilium.network.ipv4-cilium-host:172.20.38.72 io.cilium.network.ipv4-health-ip:172.20.48.223 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:40 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-12-01 23:40:25 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kubelet Update v1 2022-12-01 23:46:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-03a01a80f538d5d6b,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:40 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:46:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:40 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.46.84,},NodeAddress{Type:ExternalIP,Address:54.180.162.145,},NodeAddress{Type:Hostname,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-46-84.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-162-145.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec285e4d0d3397ed7e702d489e42654b,SystemUUID:ec285e4d-0d33-97ed-7e70-2d489e42654b,BootID:d6d1d15f-8121-4e40-b71c-e5b51419dc1b,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:48:42.721: INFO: Logging kubelet events for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:48:42.899: INFO: Logging pods the kubelet thinks is on node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:48:43.083: INFO: ebs-csi-node-lfxzl started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:48:43.083: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:48:43.083: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:48:43.083: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:48:43.083: INFO: exceed-active-deadline-57lnv started at 2022-12-01 23:48:13 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:43.083: INFO: Container c ready: true, restart count 0 Dec 1 23:48:43.083: INFO: cilium-7h664 started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:48:43.083: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:48:43.083: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:48:43.083: INFO: startup-714ba7b9-cfb1-40a5-ad4c-f7bfba82598d started at 2022-12-01 23:44:50 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:43.083: INFO: Container busybox ready: false, restart count 0 Dec 1 23:48:43.083: INFO: ss2-1 started at 2022-12-01 23:48:19 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:43.083: INFO: Container webserver ready: false, restart count 0 Dec 1 23:48:43.708: INFO: Latency metrics for node ip-172-20-46-84.ap-northeast-2.compute.internal Dec 1 23:48:43.708: INFO: Logging node info for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:48:43.884: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-54-150.ap-northeast-2.compute.internal 2a8a1d71-d00b-4259-82bd-841f96508593 32646 0 2022-12-01 23:17:10 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-54-150.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-54-150.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02d689db76221fe72"} io.cilium.network.ipv4-cilium-host:172.20.43.0 io.cilium.network.ipv4-health-ip:172.20.58.253 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-12-01 23:17:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-12-01 23:17:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-12-01 23:17:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:38 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:41:51 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02d689db76221fe72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:38 +0000 UTC,LastTransitionTime:2022-12-01 23:17:38 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:48:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:48:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:48:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:10 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:48:11 +0000 UTC,LastTransitionTime:2022-12-01 23:17:30 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.54.150,},NodeAddress{Type:ExternalIP,Address:54.180.133.96,},NodeAddress{Type:Hostname,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-54-150.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-133-96.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec20870a7ff0c6740650f49249abc430,SystemUUID:ec20870a-7ff0-c674-0650-f49249abc430,BootID:803a40bf-5805-4ae9-b933-fde612d75f1d,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 k8s.gcr.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:e7fddbaac4c3451da2365ab90bad149d32f11409738034e41e0f460927f7c276 k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.4],SizeBytes:24757245,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Dec 1 23:48:43.885: INFO: Logging kubelet events for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:48:44.062: INFO: Logging pods the kubelet thinks is on node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:48:44.242: INFO: coredns-autoscaler-6658b4bf85-92nhw started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:44.242: INFO: Container autoscaler ready: true, restart count 0 Dec 1 23:48:44.242: INFO: exceed-active-deadline-h627x started at 2022-12-01 23:48:13 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:44.242: INFO: Container c ready: true, restart count 0 Dec 1 23:48:44.242: INFO: ebs-csi-node-n5tcr started at 2022-12-01 23:17:11 +0000 UTC (0+3 container statuses recorded) Dec 1 23:48:44.243: INFO: Container ebs-plugin ready: true, restart count 0 Dec 1 23:48:44.243: INFO: Container liveness-probe ready: true, restart count 0 Dec 1 23:48:44.243: INFO: Container node-driver-registrar ready: true, restart count 0 Dec 1 23:48:44.243: INFO: replace-27832308-q8qcb started at 2022-12-01 23:48:00 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:44.243: INFO: Container c ready: true, restart count 0 Dec 1 23:48:44.243: INFO: cilium-2bxvd started at 2022-12-01 23:17:11 +0000 UTC (1+1 container statuses recorded) Dec 1 23:48:44.243: INFO: Init container clean-cilium-state ready: true, restart count 0 Dec 1 23:48:44.243: INFO: Container cilium-agent ready: true, restart count 0 Dec 1 23:48:44.243: INFO: coredns-5fcc7b6498-vcsml started at 2022-12-01 23:17:30 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:44.243: INFO: Container coredns ready: true, restart count 0 Dec 1 23:48:44.243: INFO: ss2-2 started at 2022-12-01 23:48:15 +0000 UTC (0+1 container statuses recorded) Dec 1 23:48:44.243: INFO: Container webserver ready: true, restart count 0 Dec 1 23:48:44.878: INFO: Latency metrics for node ip-172-20-54-150.ap-northeast-2.compute.internal Dec 1 23:48:44.878: INFO: Logging node info for node ip-172-20-55-57.ap-northeast-2.compute.internal Dec 1 23:48:45.054: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-57.ap-northeast-2.compute.internal 2ea0d3e0-dcd9-4562-ada8-ee5e6b866480 29483 0 2022-12-01 23:17:14 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-57.ap-northeast-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2a topology.hostpath.csi/node:ip-172-20-55-57.ap-northeast-2.compute.internal topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-02200981456309a72"} io.cilium.network.ipv4-cilium-host:172.20.51.99 io.cilium.network.ipv4-health-ip:172.20.36.120 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {kubelet Update v1 2022-12-01 23:17:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-12-01 23:17:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-12-01 23:17:44 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-12-01 23:38:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2a/i-02200981456309a72,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-12-01 23:17:44 +0000 UTC,LastTransitionTime:2022-12-01 23:17:44 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:14 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-12-01 23:46:56 +0000 UTC,LastTransitionTime:2022-12-01 23:17:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{Node