Result | FAILURE |
Tests | 66 failed / 701 succeeded |
Started | |
Elapsed | 58m0s |
Revision | master |
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(block\svolmode\)\(allowExpansion\)\]\svolume\-expand\sVerify\sif\soffline\sPVC\sexpansion\sworks$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:174 Nov 23 23:27:21.331: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc004270ce0>: { s: "pod \"pod-d51d303f-33f9-4511-8132-70e617c02080\" is not Running: timed out waiting for the condition", } pod "pod-d51d303f-33f9-4511-8132-70e617c02080" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:192from junit_22.xml
[BeforeEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Nov 23 23:22:19.476: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename volume-expand �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] Verify if offline PVC expansion works /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:174 Nov 23 23:22:20.232: INFO: Creating resource for dynamic PV Nov 23 23:22:20.232: INFO: Using claimSize:1Gi, test suite supported size:{ 1Gi}, driver(ebs.csi.aws.com) supported size:{ 1Gi} �[1mSTEP�[0m: creating a StorageClass volume-expand-2808-e2e-scx7pgp �[1mSTEP�[0m: creating a claim �[1mSTEP�[0m: Creating a pod with dynamically provisioned volume Nov 23 23:27:21.331: FAIL: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc004270ce0>: { s: "pod \"pod-d51d303f-33f9-4511-8132-70e617c02080\" is not Running: timed out waiting for the condition", } pod "pod-d51d303f-33f9-4511-8132-70e617c02080" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func4() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:192 +0x34d k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000998180) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000998180) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000998180, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 Nov 23 23:27:21.331: INFO: Deleting pod "pod-d51d303f-33f9-4511-8132-70e617c02080" in namespace "volume-expand-2808" Nov 23 23:27:21.489: INFO: Wait up to 5m0s for pod "pod-d51d303f-33f9-4511-8132-70e617c02080" to be fully deleted �[1mSTEP�[0m: Deleting pod Nov 23 23:27:21.641: INFO: Deleting pod "pod-d51d303f-33f9-4511-8132-70e617c02080" in namespace "volume-expand-2808" �[1mSTEP�[0m: Deleting pvc Nov 23 23:27:21.943: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comn9bgl" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "volume-expand-2808". �[1mSTEP�[0m: Found 6 events. Nov 23 23:27:22.400: INFO: At 2022-11-23 23:22:20 +0000 UTC - event for ebs.csi.aws.comn9bgl: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 23:27:22.400: INFO: At 2022-11-23 23:22:20 +0000 UTC - event for ebs.csi.aws.comn9bgl: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 23:27:22.400: INFO: At 2022-11-23 23:22:20 +0000 UTC - event for ebs.csi.aws.comn9bgl: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } Provisioning: External provisioner is provisioning volume for claim "volume-expand-2808/ebs.csi.aws.comn9bgl" Nov 23 23:27:22.400: INFO: At 2022-11-23 23:22:30 +0000 UTC - event for ebs.csi.aws.comn9bgl: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-2808-e2e-scx7pgp": rpc error: code = Internal desc = Could not create volume "pvc-77610927-52c9-4431-b6e7-a868e1353362": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Nov 23 23:27:22.400: INFO: At 2022-11-23 23:22:42 +0000 UTC - event for ebs.csi.aws.comn9bgl: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-2808-e2e-scx7pgp": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 23:27:22.400: INFO: At 2022-11-23 23:27:21 +0000 UTC - event for pod-d51d303f-33f9-4511-8132-70e617c02080: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "pod-d51d303f-33f9-4511-8132-70e617c02080" not found Nov 23 23:27:22.551: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 23:27:22.551: INFO: Nov 23 23:27:22.706: INFO: Logging node info for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:27:22.857: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-139.ap-northeast-1.compute.internal e52821e5-e82f-478b-be4f-6a60ba5816ce 12602 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-139.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-4383 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-35-139.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0af5c992eacc11f5f"} io.cilium.network.ipv4-cilium-host:172.20.62.73 io.cilium.network.ipv4-health-ip:172.20.63.125 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:27:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {e2e.test Update v1 2022-11-23 23:27:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0af5c992eacc11f5f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:51 +0000 UTC,LastTransitionTime:2022-11-23 23:18:51 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:27:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.139,},NodeAddress{Type:ExternalIP,Address:13.231.193.129,},NodeAddress{Type:Hostname,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-231-193-129.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec218b5e615d578e61275cbc46e5a775,SystemUUID:ec218b5e-615d-578e-6127-5cbc46e5a775,BootID:15a3520e-c72a-46fe-a3d6-be039aa283b7,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:27:22.858: INFO: Logging kubelet events for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:27:23.011: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:27:23.177: INFO: test-container-pod started at 2022-11-23 23:27:12 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:23.177: INFO: Container webserver ready: true, restart count 0 Nov 23 23:27:23.177: INFO: cilium-dw7qk started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:23.177: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:27:23.177: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:27:23.177: INFO: netserver-0 started at 2022-11-23 23:26:50 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:23.177: INFO: Container webserver ready: true, restart count 0 Nov 23 23:27:23.177: INFO: test-container-pod started at 2022-11-23 23:27:11 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:23.177: INFO: Container webserver ready: true, restart count 0 Nov 23 23:27:23.177: INFO: netserver-0 started at 2022-11-23 23:26:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:23.177: INFO: Container webserver ready: true, restart count 0 Nov 23 23:27:23.177: INFO: ebs-csi-node-vbdc7 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:23.177: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:23.177: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:23.177: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:27:23.177: INFO: nodeport-test-89rgk started at 2022-11-23 23:27:10 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:23.177: INFO: Container nodeport-test ready: true, restart count 0 Nov 23 23:27:23.177: INFO: hairpin started at 2022-11-23 23:27:19 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:23.177: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:27:23.177: INFO: metadata-volume-c1b8e007-0c4b-4b6c-9aec-ce455c4610f6 started at 2022-11-23 23:27:22 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:23.177: INFO: Container client-container ready: false, restart count 0 Nov 23 23:27:23.697: INFO: Latency metrics for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:27:23.697: INFO: Logging node info for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:27:23.849: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-241.ap-northeast-1.compute.internal 86ed4d71-4560-48af-9e88-d8d85c882f8b 6905 0 2022-11-23 23:16:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-241.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-027602f60f12db3f0"} io.cilium.network.ipv4-cilium-host:172.20.42.20 io.cilium.network.ipv4-health-ip:172.20.32.68 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 23:16:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-11-23 23:16:55 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-11-23 23:17:04 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:17:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 23:18:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:18:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-027602f60f12db3f0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892301824 0} {<nil>} 3801076Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787444224 0} {<nil>} 3698676Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:42 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:17:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.241,},NodeAddress{Type:ExternalIP,Address:54.238.84.104,},NodeAddress{Type:Hostname,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-84-104.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2adfc78fc274df02e3c085caa34395,SystemUUID:ec2adfc7-8fc2-74df-02e3-c085caa34395,BootID:e3997507-167a-4ec7-a6ec-7f97c5d886f5,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42904862,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41478108,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955301,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:27:23.849: INFO: Logging kubelet events for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:27:24.005: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:27:24.163: INFO: etcd-manager-events-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:24.163: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:27:24.163: INFO: cilium-qvdj2 started at 2022-11-23 23:17:02 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:24.163: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:27:24.163: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 23:27:24.163: INFO: cilium-operator-587dd66f87-j2km7 started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:24.163: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 23:27:24.163: INFO: ebs-csi-controller-764b64d679-r8ttz started at 2022-11-23 23:17:03 +0000 UTC (0+5 container statuses recorded) Nov 23 23:27:24.163: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:27:24.163: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:27:24.163: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:27:24.163: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:24.163: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:24.163: INFO: ebs-csi-node-94jwj started at 2022-11-23 23:17:02 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:24.163: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:24.163: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:24.163: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:27:24.163: INFO: kops-controller-fznlr started at 2022-11-23 23:17:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:24.163: INFO: Container kops-controller ready: true, restart count 0 Nov 23 23:27:24.163: INFO: dns-controller-845449865f-zwg9n started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:24.163: INFO: Container dns-controller ready: true, restart count 0 Nov 23 23:27:24.163: INFO: kube-controller-manager-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:24.163: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 23:27:24.163: INFO: kube-scheduler-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:24.163: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 23:27:24.163: INFO: etcd-manager-main-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:24.163: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:27:24.163: INFO: kube-apiserver-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+2 container statuses recorded) Nov 23 23:27:24.163: INFO: Container healthcheck ready: true, restart count 0 Nov 23 23:27:24.163: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 23:27:24.685: INFO: Latency metrics for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:27:24.685: INFO: Logging node info for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:27:24.838: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-42-64.ap-northeast-1.compute.internal 45916c04-689e-47fa-b6fb-bdbad0784fcd 9817 0 2022-11-23 23:18:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-42-64.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-42-64.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-049ec10a59a1d20e4"} io.cilium.network.ipv4-cilium-host:172.20.51.104 io.cilium.network.ipv4-health-ip:172.20.34.63 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:22:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-049ec10a59a1d20e4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:56 +0000 UTC,LastTransitionTime:2022-11-23 23:18:56 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.64,},NodeAddress{Type:ExternalIP,Address:54.238.96.164,},NodeAddress{Type:Hostname,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-96-164.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25c6eb725883a2c2916889bd12be88,SystemUUID:ec25c6eb-7258-83a2-c291-6889bd12be88,BootID:8214bf1e-ee6e-4136-8902-31d30749be64,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:27:24.838: INFO: Logging kubelet events for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:27:24.992: INFO: Logging pods the kubelet thinks is on node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:27:25.151: INFO: ebs-csi-node-fq5pw started at 2022-11-23 23:18:29 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:25.151: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:25.151: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:25.151: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:27:25.151: INFO: cilium-jjmjs started at 2022-11-23 23:18:29 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:25.151: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:27:25.151: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:27:25.151: INFO: hostexec-ip-172-20-42-64.ap-northeast-1.compute.internal-v6wzb started at 2022-11-23 23:26:46 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:25.151: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:27:25.151: INFO: pod-subpath-test-preprovisionedpv-2d76 started at 2022-11-23 23:27:03 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:25.151: INFO: Init container init-volume-preprovisionedpv-2d76 ready: true, restart count 0 Nov 23 23:27:25.151: INFO: Container test-container-subpath-preprovisionedpv-2d76 ready: true, restart count 0 Nov 23 23:27:25.151: INFO: hostexec-ip-172-20-42-64.ap-northeast-1.compute.internal-d97wc started at 2022-11-23 23:26:51 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:25.151: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:27:25.151: INFO: coredns-5fcc7b6498-s4jjd started at 2022-11-23 23:19:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:25.151: INFO: Container coredns ready: true, restart count 0 Nov 23 23:27:25.151: INFO: foo-tz8t2 started at 2022-11-23 23:26:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:25.151: INFO: Container c ready: true, restart count 0 Nov 23 23:27:25.663: INFO: Latency metrics for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:27:25.663: INFO: Logging node info for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:27:25.814: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-200.ap-northeast-1.compute.internal 8f32a357-4fd8-4fcd-ab21-8b69dbd931d6 7902 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-200.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d4c764fea6a24b23"} io.cilium.network.ipv4-cilium-host:172.20.61.17 io.cilium.network.ipv4-health-ip:172.20.60.108 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:50 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:24:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0d4c764fea6a24b23,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:50 +0000 UTC,LastTransitionTime:2022-11-23 23:18:50 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:24:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.200,},NodeAddress{Type:ExternalIP,Address:3.113.30.221,},NodeAddress{Type:Hostname,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-113-30-221.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ef1c5e899b01a14fef22e61702fe5,SystemUUID:ec2ef1c5-e899-b01a-14fe-f22e61702fe5,BootID:021f6d58-0800-4a46-b65e-3513c48dff08,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:27:25.815: INFO: Logging kubelet events for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:27:25.968: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:27:26.124: INFO: ebs-csi-node-dl575 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:26.124: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:26.124: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:26.124: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:27:26.124: INFO: local-injector started at 2022-11-23 23:27:19 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:26.124: INFO: Container local-injector ready: true, restart count 0 Nov 23 23:27:26.124: INFO: hostexec-ip-172-20-55-200.ap-northeast-1.compute.internal-wp6t8 started at 2022-11-23 23:27:07 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:26.124: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:27:26.124: INFO: cilium-2lrb9 started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:26.124: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:27:26.124: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:27:26.124: INFO: coredns-autoscaler-6658b4bf85-gdgxf started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:26.124: INFO: Container autoscaler ready: true, restart count 0 Nov 23 23:27:26.124: INFO: coredns-5fcc7b6498-ll578 started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:26.124: INFO: Container coredns ready: true, restart count 0 Nov 23 23:27:26.124: INFO: pod-ephm-test-projected-vkdw started at 2022-11-23 23:25:27 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:26.124: INFO: Container test-container-subpath-projected-vkdw ready: false, restart count 0 Nov 23 23:27:26.633: INFO: Latency metrics for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:27:26.633: INFO: Logging node info for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:27:26.785: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-63-80.ap-northeast-1.compute.internal 1d4cdbf2-ca1f-431f-b562-a0bd0f489514 12148 0 2022-11-23 23:18:44 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-63-80.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-63-80.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-081735fbd2bce39f2"} io.cilium.network.ipv4-cilium-host:172.20.50.140 io.cilium.network.ipv4-health-ip:172.20.42.94 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:19:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:19:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:26:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-081735fbd2bce39f2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:19:12 +0000 UTC,LastTransitionTime:2022-11-23 23:19:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:26:24 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:26:24 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:26:24 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:26:24 +0000 UTC,LastTransitionTime:2022-11-23 23:19:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.63.80,},NodeAddress{Type:ExternalIP,Address:13.112.241.172,},NodeAddress{Type:Hostname,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-112-241-172.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29233394293c3c1bf3f4359c382b54,SystemUUID:ec292333-9429-3c3c-1bf3-f4359c382b54,BootID:05304de1-80b3-439c-9eeb-74d1d733f407,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:27:26.786: INFO: Logging kubelet events for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:27:26.945: INFO: Logging pods the kubelet thinks is on node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:27:27.105: INFO: csi-mockplugin-0 started at 2022-11-23 23:26:19 +0000 UTC (0+4 container statuses recorded) Nov 23 23:27:27.105: INFO: Container busybox ready: true, restart count 0 Nov 23 23:27:27.105: INFO: Container csi-provisioner ready: false, restart count 2 Nov 23 23:27:27.105: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:27:27.105: INFO: Container mock ready: true, restart count 0 Nov 23 23:27:27.105: INFO: nodeport-test-jphzg started at 2022-11-23 23:27:10 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:27.105: INFO: Container nodeport-test ready: true, restart count 0 Nov 23 23:27:27.105: INFO: foo-d8nnh started at 2022-11-23 23:26:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:27.105: INFO: Container c ready: true, restart count 0 Nov 23 23:27:27.106: INFO: cilium-dgm5r started at 2022-11-23 23:18:44 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:27.106: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:27:27.106: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:27:27.106: INFO: execpodpr7qn started at 2022-11-23 23:27:13 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:27.106: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:27:27.106: INFO: ebs-csi-node-hlgh2 started at 2022-11-23 23:18:44 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:27.106: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:27.106: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:27.106: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:27:27.106: INFO: startup-90ffdccd-1fcc-4f83-8ed9-18ea687b1842 started at 2022-11-23 23:27:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:27.106: INFO: Container busybox ready: false, restart count 0 Nov 23 23:27:27.656: INFO: Latency metrics for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:27:27.657: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volume-expand-2808" for this suite.
Find pod-d51d303f-33f9-4511-8132-70e617c02080 mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(block\svolmode\)\(allowExpansion\)\]\svolume\-expand\sshould\sresize\svolume\swhen\sPVC\sis\sedited\swhile\spod\sis\susing\sit$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:246 Nov 23 23:29:01.532: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc0053f1680>: { s: "pod \"pod-ca4853fb-ce70-40ad-9e9e-629b1e16c6a5\" is not Running: timed out waiting for the condition", } pod "pod-ca4853fb-ce70-40ad-9e9e-629b1e16c6a5" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:268from junit_11.xml
[BeforeEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Nov 23 23:23:59.627: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename volume-expand �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should resize volume when PVC is edited while pod is using it /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:246 Nov 23 23:24:00.419: INFO: Creating resource for dynamic PV Nov 23 23:24:00.419: INFO: Using claimSize:1Gi, test suite supported size:{ 1Gi}, driver(ebs.csi.aws.com) supported size:{ 1Gi} �[1mSTEP�[0m: creating a StorageClass volume-expand-2797-e2e-scbc9gf �[1mSTEP�[0m: creating a claim �[1mSTEP�[0m: Creating a pod with dynamically provisioned volume Nov 23 23:29:01.532: FAIL: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc0053f1680>: { s: "pod \"pod-ca4853fb-ce70-40ad-9e9e-629b1e16c6a5\" is not Running: timed out waiting for the condition", } pod "pod-ca4853fb-ce70-40ad-9e9e-629b1e16c6a5" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func5() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:268 +0x3d0 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000bfe300) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000bfe300) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000bfe300, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 Nov 23 23:29:01.532: INFO: Deleting pod "pod-ca4853fb-ce70-40ad-9e9e-629b1e16c6a5" in namespace "volume-expand-2797" Nov 23 23:29:01.693: INFO: Wait up to 5m0s for pod "pod-ca4853fb-ce70-40ad-9e9e-629b1e16c6a5" to be fully deleted �[1mSTEP�[0m: Deleting pod Nov 23 23:29:01.851: INFO: Deleting pod "pod-ca4853fb-ce70-40ad-9e9e-629b1e16c6a5" in namespace "volume-expand-2797" �[1mSTEP�[0m: Deleting pvc Nov 23 23:29:02.168: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comkcclx" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "volume-expand-2797". �[1mSTEP�[0m: Found 6 events. Nov 23 23:29:02.645: INFO: At 2022-11-23 23:24:00 +0000 UTC - event for ebs.csi.aws.comkcclx: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 23:29:02.645: INFO: At 2022-11-23 23:24:00 +0000 UTC - event for ebs.csi.aws.comkcclx: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 23:29:02.645: INFO: At 2022-11-23 23:24:00 +0000 UTC - event for ebs.csi.aws.comkcclx: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } Provisioning: External provisioner is provisioning volume for claim "volume-expand-2797/ebs.csi.aws.comkcclx" Nov 23 23:29:02.645: INFO: At 2022-11-23 23:24:11 +0000 UTC - event for ebs.csi.aws.comkcclx: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-2797-e2e-scbc9gf": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 23:29:02.645: INFO: At 2022-11-23 23:27:28 +0000 UTC - event for ebs.csi.aws.comkcclx: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-2797-e2e-scbc9gf": rpc error: code = Internal desc = Could not create volume "pvc-dd071b53-3afd-4d37-a46d-bc31bc25b84c": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Nov 23 23:29:02.645: INFO: At 2022-11-23 23:29:01 +0000 UTC - event for pod-ca4853fb-ce70-40ad-9e9e-629b1e16c6a5: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "pod-ca4853fb-ce70-40ad-9e9e-629b1e16c6a5" not found Nov 23 23:29:02.803: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 23:29:02.803: INFO: Nov 23 23:29:02.964: INFO: Logging node info for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:29:03.122: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-139.ap-northeast-1.compute.internal e52821e5-e82f-478b-be4f-6a60ba5816ce 14113 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-139.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-4383 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-35-139.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-4950":"csi-mock-csi-mock-volumes-4950","ebs.csi.aws.com":"i-0af5c992eacc11f5f"} io.cilium.network.ipv4-cilium-host:172.20.62.73 io.cilium.network.ipv4-health-ip:172.20.63.125 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-11-23 23:27:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:27:56 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 23:28:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0af5c992eacc11f5f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:51 +0000 UTC,LastTransitionTime:2022-11-23 23:18:51 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:28:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:28:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:28:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:28:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.139,},NodeAddress{Type:ExternalIP,Address:13.231.193.129,},NodeAddress{Type:Hostname,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-231-193-129.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec218b5e615d578e61275cbc46e5a775,SystemUUID:ec218b5e-615d-578e-6127-5cbc46e5a775,BootID:15a3520e-c72a-46fe-a3d6-be039aa283b7,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-4950^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-4950^4,DevicePath:,},},Config:nil,},} Nov 23 23:29:03.123: INFO: Logging kubelet events for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:29:03.314: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:29:03.641: INFO: pvc-volume-tester-cjrdx started at 2022-11-23 23:27:55 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:03.641: INFO: Container volume-tester ready: true, restart count 0 Nov 23 23:29:03.641: INFO: frontend-74dc45c94b-b49rc started at 2022-11-23 23:28:55 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:03.641: INFO: Container guestbook-frontend ready: true, restart count 0 Nov 23 23:29:03.641: INFO: pod-service-account-defaultsa-nomountspec started at 2022-11-23 23:28:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:03.641: INFO: Container token-test ready: true, restart count 0 Nov 23 23:29:03.641: INFO: pod-service-account-nomountsa-nomountspec started at 2022-11-23 23:28:58 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:03.641: INFO: Container token-test ready: false, restart count 0 Nov 23 23:29:03.641: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:27:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:03.641: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:29:03.641: INFO: cilium-dw7qk started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:29:03.641: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:29:03.641: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:29:03.641: INFO: rs-nk4kz started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:03.641: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:03.641: INFO: pod-service-account-mountsa-mountspec started at 2022-11-23 23:28:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:03.641: INFO: Container token-test ready: true, restart count 0 Nov 23 23:29:03.641: INFO: ebs-csi-node-vbdc7 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:29:03.641: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:29:03.641: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:29:03.641: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:29:03.641: INFO: pod-service-account-nomountsa-mountspec started at 2022-11-23 23:28:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:03.641: INFO: Container token-test ready: true, restart count 0 Nov 23 23:29:03.641: INFO: rs-5lh9m started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:03.641: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:03.641: INFO: pod-service-account-mountsa-nomountspec started at 2022-11-23 23:28:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:03.641: INFO: Container token-test ready: true, restart count 0 Nov 23 23:29:03.641: INFO: csi-mockplugin-0 started at 2022-11-23 23:27:48 +0000 UTC (0+3 container statuses recorded) Nov 23 23:29:03.641: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:29:03.641: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:29:03.641: INFO: Container mock ready: true, restart count 0 Nov 23 23:29:03.641: INFO: rs-km877 started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:03.641: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:04.228: INFO: Latency metrics for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:29:04.228: INFO: Logging node info for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:29:04.390: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-241.ap-northeast-1.compute.internal 86ed4d71-4560-48af-9e88-d8d85c882f8b 6905 0 2022-11-23 23:16:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-241.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-027602f60f12db3f0"} io.cilium.network.ipv4-cilium-host:172.20.42.20 io.cilium.network.ipv4-health-ip:172.20.32.68 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 23:16:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-11-23 23:16:55 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-11-23 23:17:04 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:17:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 23:18:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:18:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-027602f60f12db3f0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892301824 0} {<nil>} 3801076Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787444224 0} {<nil>} 3698676Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:42 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:17:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.241,},NodeAddress{Type:ExternalIP,Address:54.238.84.104,},NodeAddress{Type:Hostname,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-84-104.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2adfc78fc274df02e3c085caa34395,SystemUUID:ec2adfc7-8fc2-74df-02e3-c085caa34395,BootID:e3997507-167a-4ec7-a6ec-7f97c5d886f5,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42904862,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41478108,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955301,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:29:04.390: INFO: Logging kubelet events for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:29:04.551: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:29:04.714: INFO: kube-apiserver-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+2 container statuses recorded) Nov 23 23:29:04.714: INFO: Container healthcheck ready: true, restart count 0 Nov 23 23:29:04.714: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 23:29:04.714: INFO: ebs-csi-node-94jwj started at 2022-11-23 23:17:02 +0000 UTC (0+3 container statuses recorded) Nov 23 23:29:04.714: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:29:04.714: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:29:04.714: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:29:04.714: INFO: kops-controller-fznlr started at 2022-11-23 23:17:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:04.714: INFO: Container kops-controller ready: true, restart count 0 Nov 23 23:29:04.714: INFO: dns-controller-845449865f-zwg9n started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:04.714: INFO: Container dns-controller ready: true, restart count 0 Nov 23 23:29:04.714: INFO: kube-controller-manager-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:04.714: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 23:29:04.714: INFO: kube-scheduler-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:04.714: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 23:29:04.714: INFO: etcd-manager-main-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:04.714: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:29:04.714: INFO: ebs-csi-controller-764b64d679-r8ttz started at 2022-11-23 23:17:03 +0000 UTC (0+5 container statuses recorded) Nov 23 23:29:04.714: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:29:04.714: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:29:04.714: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:29:04.714: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:29:04.714: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:29:04.714: INFO: etcd-manager-events-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:04.714: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:29:04.714: INFO: cilium-qvdj2 started at 2022-11-23 23:17:02 +0000 UTC (1+1 container statuses recorded) Nov 23 23:29:04.714: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:29:04.714: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 23:29:04.714: INFO: cilium-operator-587dd66f87-j2km7 started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:04.714: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 23:29:05.268: INFO: Latency metrics for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:29:05.268: INFO: Logging node info for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:29:05.427: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-42-64.ap-northeast-1.compute.internal 45916c04-689e-47fa-b6fb-bdbad0784fcd 9817 0 2022-11-23 23:18:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-42-64.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-42-64.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-049ec10a59a1d20e4"} io.cilium.network.ipv4-cilium-host:172.20.51.104 io.cilium.network.ipv4-health-ip:172.20.34.63 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:22:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-049ec10a59a1d20e4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:56 +0000 UTC,LastTransitionTime:2022-11-23 23:18:56 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.64,},NodeAddress{Type:ExternalIP,Address:54.238.96.164,},NodeAddress{Type:Hostname,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-96-164.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25c6eb725883a2c2916889bd12be88,SystemUUID:ec25c6eb-7258-83a2-c291-6889bd12be88,BootID:8214bf1e-ee6e-4136-8902-31d30749be64,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:29:05.427: INFO: Logging kubelet events for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:29:05.589: INFO: Logging pods the kubelet thinks is on node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:29:05.752: INFO: inline-volume-c2zwr started at 2022-11-23 23:27:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:05.752: INFO: Container volume-tester ready: false, restart count 0 Nov 23 23:29:05.752: INFO: inline-volume-xhkps started at 2022-11-23 23:28:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:05.752: INFO: Container volume-tester ready: false, restart count 0 Nov 23 23:29:05.752: INFO: agnhost-replica-6987b74bb4-hf28s started at 2022-11-23 23:28:56 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:05.752: INFO: Container replica ready: true, restart count 0 Nov 23 23:29:05.752: INFO: cilium-jjmjs started at 2022-11-23 23:18:29 +0000 UTC (1+1 container statuses recorded) Nov 23 23:29:05.752: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:29:05.752: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:29:05.752: INFO: rs-gfcss started at 2022-11-23 23:29:04 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:05.752: INFO: Container donothing ready: false, restart count 0 Nov 23 23:29:05.752: INFO: startup-73da1a16-b444-4ab8-955f-74bbd9876a25 started at 2022-11-23 23:28:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:05.752: INFO: Container busybox ready: true, restart count 0 Nov 23 23:29:05.752: INFO: coredns-5fcc7b6498-s4jjd started at 2022-11-23 23:19:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:05.752: INFO: Container coredns ready: true, restart count 0 Nov 23 23:29:05.752: INFO: ebs-csi-node-fq5pw started at 2022-11-23 23:18:29 +0000 UTC (0+3 container statuses recorded) Nov 23 23:29:05.752: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:29:05.752: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:29:05.752: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:29:05.752: INFO: rs-zkqnm started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:05.752: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:05.752: INFO: frontend-74dc45c94b-mxxhx started at 2022-11-23 23:28:55 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:05.752: INFO: Container guestbook-frontend ready: true, restart count 0 Nov 23 23:29:06.341: INFO: Latency metrics for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:29:06.341: INFO: Logging node info for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:29:06.500: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-200.ap-northeast-1.compute.internal 8f32a357-4fd8-4fcd-ab21-8b69dbd931d6 13785 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-200.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-6944":"csi-mock-csi-mock-volumes-6944","ebs.csi.aws.com":"i-0d4c764fea6a24b23"} io.cilium.network.ipv4-cilium-host:172.20.61.17 io.cilium.network.ipv4-health-ip:172.20.60.108 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:50 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:24:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0d4c764fea6a24b23,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:50 +0000 UTC,LastTransitionTime:2022-11-23 23:18:50 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.200,},NodeAddress{Type:ExternalIP,Address:3.113.30.221,},NodeAddress{Type:Hostname,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-113-30-221.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ef1c5e899b01a14fef22e61702fe5,SystemUUID:ec2ef1c5-e899-b01a-14fe-f22e61702fe5,BootID:021f6d58-0800-4a46-b65e-3513c48dff08,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:29:06.501: INFO: Logging kubelet events for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:29:06.661: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:29:06.824: INFO: rs-dthbz started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:06.824: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:06.824: INFO: csi-mockplugin-0 started at 2022-11-23 23:27:49 +0000 UTC (0+3 container statuses recorded) Nov 23 23:29:06.824: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:29:06.824: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:29:06.824: INFO: Container mock ready: true, restart count 0 Nov 23 23:29:06.824: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:27:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:06.824: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:29:06.824: INFO: pod-0 started at 2022-11-23 23:28:01 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:06.824: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:06.824: INFO: cilium-2lrb9 started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:29:06.824: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:29:06.824: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:29:06.824: INFO: coredns-autoscaler-6658b4bf85-gdgxf started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:06.824: INFO: Container autoscaler ready: true, restart count 0 Nov 23 23:29:06.824: INFO: coredns-5fcc7b6498-ll578 started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:06.824: INFO: Container coredns ready: true, restart count 0 Nov 23 23:29:06.824: INFO: pod-2 started at 2022-11-23 23:28:01 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:06.824: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:06.824: INFO: startup-script started at 2022-11-23 23:28:04 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:06.824: INFO: Container startup-script ready: true, restart count 0 Nov 23 23:29:06.824: INFO: ebs-csi-node-dl575 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:29:06.824: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:29:06.824: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:29:06.824: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:29:06.824: INFO: pvc-volume-tester-wrf6r started at 2022-11-23 23:27:55 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:06.824: INFO: Container volume-tester ready: true, restart count 0 Nov 23 23:29:06.824: INFO: rs-mg6vl started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:06.824: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:07.385: INFO: Latency metrics for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:29:07.386: INFO: Logging node info for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:29:07.545: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-63-80.ap-northeast-1.compute.internal 1d4cdbf2-ca1f-431f-b562-a0bd0f489514 15090 0 2022-11-23 23:18:44 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-63-80.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-63-80.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-081735fbd2bce39f2"} io.cilium.network.ipv4-cilium-host:172.20.50.140 io.cilium.network.ipv4-health-ip:172.20.42.94 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:19:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:19:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:26:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-081735fbd2bce39f2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:19:12 +0000 UTC,LastTransitionTime:2022-11-23 23:19:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:28:44 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:28:44 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:28:44 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:28:44 +0000 UTC,LastTransitionTime:2022-11-23 23:19:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.63.80,},NodeAddress{Type:ExternalIP,Address:13.112.241.172,},NodeAddress{Type:Hostname,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-112-241-172.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29233394293c3c1bf3f4359c382b54,SystemUUID:ec292333-9429-3c3c-1bf3-f4359c382b54,BootID:05304de1-80b3-439c-9eeb-74d1d733f407,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:29:07.545: INFO: Logging kubelet events for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:29:07.707: INFO: Logging pods the kubelet thinks is on node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:29:07.881: INFO: pod-projected-configmaps-560b66d7-f380-4190-bad4-60cf6426eb20 started at <nil> (0+0 container statuses recorded) Nov 23 23:29:07.881: INFO: ebs-csi-node-hlgh2 started at 2022-11-23 23:18:44 +0000 UTC (0+3 container statuses recorded) Nov 23 23:29:07.881: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:29:07.881: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:29:07.881: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:29:07.881: INFO: sample-webhook-deployment-5d5bc8ffd5-q8cf7 started at 2022-11-23 23:29:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:07.881: INFO: Container sample-webhook ready: false, restart count 0 Nov 23 23:29:07.881: INFO: pod-service-account-defaultsa-mountspec started at 2022-11-23 23:28:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:07.881: INFO: Container token-test ready: false, restart count 0 Nov 23 23:29:07.881: INFO: frontend-74dc45c94b-wzt7q started at 2022-11-23 23:28:55 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:07.881: INFO: Container guestbook-frontend ready: true, restart count 0 Nov 23 23:29:07.881: INFO: hostexec-ip-172-20-63-80.ap-northeast-1.compute.internal-7k8z7 started at <nil> (0+0 container statuses recorded) Nov 23 23:29:07.881: INFO: pod-service-account-defaultsa started at 2022-11-23 23:28:56 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:07.881: INFO: Container token-test ready: false, restart count 0 Nov 23 23:29:07.881: INFO: alpine-nnp-false-c013d00a-fa87-43ee-897a-7a5a250c6770 started at 2022-11-23 23:28:59 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:07.881: INFO: Container alpine-nnp-false-c013d00a-fa87-43ee-897a-7a5a250c6770 ready: false, restart count 0 Nov 23 23:29:07.881: INFO: pod-logs-websocket-efa8f17c-4ebe-4b90-8a38-1592a7ce896c started at 2022-11-23 23:28:37 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:07.881: INFO: Container main ready: true, restart count 0 Nov 23 23:29:07.881: INFO: rs-fv8pd started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:07.881: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:07.881: INFO: rs-scn49 started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:07.881: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:07.881: INFO: agnhost-primary-58467dcb76-jz5jh started at 2022-11-23 23:28:55 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:07.881: INFO: Container primary ready: true, restart count 0 Nov 23 23:29:07.881: INFO: rs-wrj5z started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:07.881: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:07.881: INFO: agnhost-replica-6987b74bb4-r5zbl started at 2022-11-23 23:28:56 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:07.881: INFO: Container replica ready: true, restart count 0 Nov 23 23:29:07.881: INFO: pfpod started at <nil> (0+0 container statuses recorded) Nov 23 23:29:07.881: INFO: cilium-dgm5r started at 2022-11-23 23:18:44 +0000 UTC (1+1 container statuses recorded) Nov 23 23:29:07.881: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:29:07.881: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:29:07.881: INFO: boom-server started at 2022-11-23 23:27:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:07.881: INFO: Container boom-server ready: true, restart count 0 Nov 23 23:29:07.881: INFO: pod-1 started at 2022-11-23 23:28:01 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:07.881: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:08.700: INFO: Latency metrics for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:29:08.700: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volume-expand-2797" for this suite.
Find pod-ca4853fb-ce70-40ad-9e9e-629b1e16c6a5 mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(block\svolmode\)\]\svolumeMode\sshould\snot\smount\s\/\smap\sunused\svolumes\sin\sa\spod\s\[LinuxOnly\]$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:352 Nov 23 23:27:47.495: Unexpected error: <*errors.errorString | 0xc000248290>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:385from junit_14.xml
{"msg":"PASSED [sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should support non-existent path","total":-1,"completed":3,"skipped":26,"failed":0} [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Nov 23 23:22:45.612: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename volumemode �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should not mount / map unused volumes in a pod [LinuxOnly] /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:352 Nov 23 23:22:46.393: INFO: Creating resource for dynamic PV Nov 23 23:22:46.393: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass volumemode-1607-e2e-scb6fdm �[1mSTEP�[0m: creating a claim �[1mSTEP�[0m: Creating pod Nov 23 23:27:47.495: FAIL: Unexpected error: <*errors.errorString | 0xc000248290>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeModeTestSuite).DefineTests.func7() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:385 +0x6c5 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc00046f680) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc00046f680) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc00046f680, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 Nov 23 23:27:47.495: INFO: Deleting pod "pod-04a3d9a6-bc7a-4f63-81c2-ecb29b309af0" in namespace "volumemode-1607" Nov 23 23:27:47.658: INFO: Wait up to 5m0s for pod "pod-04a3d9a6-bc7a-4f63-81c2-ecb29b309af0" to be fully deleted �[1mSTEP�[0m: Deleting pvc Nov 23 23:27:47.972: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comd748c" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (block volmode)] volumeMode /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "volumemode-1607". �[1mSTEP�[0m: Found 6 events. Nov 23 23:27:48.446: INFO: At 2022-11-23 23:22:46 +0000 UTC - event for ebs.csi.aws.comd748c: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 23:27:48.446: INFO: At 2022-11-23 23:22:46 +0000 UTC - event for ebs.csi.aws.comd748c: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 23:27:48.446: INFO: At 2022-11-23 23:22:46 +0000 UTC - event for ebs.csi.aws.comd748c: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } Provisioning: External provisioner is provisioning volume for claim "volumemode-1607/ebs.csi.aws.comd748c" Nov 23 23:27:48.446: INFO: At 2022-11-23 23:22:57 +0000 UTC - event for ebs.csi.aws.comd748c: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "volumemode-1607-e2e-scb6fdm": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 23:27:48.446: INFO: At 2022-11-23 23:24:18 +0000 UTC - event for ebs.csi.aws.comd748c: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "volumemode-1607-e2e-scb6fdm": rpc error: code = Internal desc = Could not create volume "pvc-fa1b330c-82fb-4c6a-a3ee-5dcdd1aeb964": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Nov 23 23:27:48.446: INFO: At 2022-11-23 23:27:47 +0000 UTC - event for pod-04a3d9a6-bc7a-4f63-81c2-ecb29b309af0: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "pod-04a3d9a6-bc7a-4f63-81c2-ecb29b309af0" not found Nov 23 23:27:48.602: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 23:27:48.602: INFO: Nov 23 23:27:48.764: INFO: Logging node info for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:27:48.921: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-139.ap-northeast-1.compute.internal e52821e5-e82f-478b-be4f-6a60ba5816ce 12602 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-139.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-4383 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-35-139.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0af5c992eacc11f5f"} io.cilium.network.ipv4-cilium-host:172.20.62.73 io.cilium.network.ipv4-health-ip:172.20.63.125 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:27:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {e2e.test Update v1 2022-11-23 23:27:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0af5c992eacc11f5f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:51 +0000 UTC,LastTransitionTime:2022-11-23 23:18:51 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:27:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.139,},NodeAddress{Type:ExternalIP,Address:13.231.193.129,},NodeAddress{Type:Hostname,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-231-193-129.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec218b5e615d578e61275cbc46e5a775,SystemUUID:ec218b5e-615d-578e-6127-5cbc46e5a775,BootID:15a3520e-c72a-46fe-a3d6-be039aa283b7,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:27:48.921: INFO: Logging kubelet events for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:27:49.080: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:27:49.246: INFO: ebs-csi-node-vbdc7 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:49.247: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:49.247: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:49.247: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:27:49.247: INFO: simpletest.rc-htb2w started at 2022-11-23 23:27:31 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:49.247: INFO: Container nginx ready: true, restart count 0 Nov 23 23:27:49.247: INFO: csi-mockplugin-0 started at 2022-11-23 23:27:48 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:49.247: INFO: Container csi-provisioner ready: false, restart count 0 Nov 23 23:27:49.247: INFO: Container driver-registrar ready: false, restart count 0 Nov 23 23:27:49.247: INFO: Container mock ready: false, restart count 0 Nov 23 23:27:49.247: INFO: simpletest.rc-j7bh2 started at 2022-11-23 23:27:31 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:49.247: INFO: Container nginx ready: true, restart count 0 Nov 23 23:27:49.247: INFO: simpletest.rc-p5gfw started at 2022-11-23 23:27:31 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:49.247: INFO: Container nginx ready: true, restart count 0 Nov 23 23:27:49.247: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:27:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:49.247: INFO: Container csi-attacher ready: false, restart count 0 Nov 23 23:27:49.247: INFO: cilium-dw7qk started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:49.247: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:27:49.247: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:27:49.839: INFO: Latency metrics for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:27:49.839: INFO: Logging node info for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:27:49.996: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-241.ap-northeast-1.compute.internal 86ed4d71-4560-48af-9e88-d8d85c882f8b 6905 0 2022-11-23 23:16:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-241.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-027602f60f12db3f0"} io.cilium.network.ipv4-cilium-host:172.20.42.20 io.cilium.network.ipv4-health-ip:172.20.32.68 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 23:16:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-11-23 23:16:55 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-11-23 23:17:04 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:17:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 23:18:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:18:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-027602f60f12db3f0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892301824 0} {<nil>} 3801076Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787444224 0} {<nil>} 3698676Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:42 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:17:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.241,},NodeAddress{Type:ExternalIP,Address:54.238.84.104,},NodeAddress{Type:Hostname,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-84-104.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2adfc78fc274df02e3c085caa34395,SystemUUID:ec2adfc7-8fc2-74df-02e3-c085caa34395,BootID:e3997507-167a-4ec7-a6ec-7f97c5d886f5,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42904862,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41478108,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955301,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:27:49.996: INFO: Logging kubelet events for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:27:50.156: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:27:50.472: INFO: kube-controller-manager-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:50.472: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 23:27:50.472: INFO: kube-scheduler-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:50.472: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 23:27:50.472: INFO: etcd-manager-main-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:50.472: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:27:50.472: INFO: kube-apiserver-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+2 container statuses recorded) Nov 23 23:27:50.472: INFO: Container healthcheck ready: true, restart count 0 Nov 23 23:27:50.472: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 23:27:50.472: INFO: ebs-csi-node-94jwj started at 2022-11-23 23:17:02 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:50.472: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:50.472: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:50.472: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:27:50.472: INFO: kops-controller-fznlr started at 2022-11-23 23:17:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:50.472: INFO: Container kops-controller ready: true, restart count 0 Nov 23 23:27:50.472: INFO: dns-controller-845449865f-zwg9n started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:50.472: INFO: Container dns-controller ready: true, restart count 0 Nov 23 23:27:50.472: INFO: etcd-manager-events-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:50.472: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:27:50.472: INFO: cilium-qvdj2 started at 2022-11-23 23:17:02 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:50.472: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:27:50.472: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 23:27:50.472: INFO: cilium-operator-587dd66f87-j2km7 started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:50.472: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 23:27:50.472: INFO: ebs-csi-controller-764b64d679-r8ttz started at 2022-11-23 23:17:03 +0000 UTC (0+5 container statuses recorded) Nov 23 23:27:50.472: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:27:50.472: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:27:50.472: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:27:50.472: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:50.472: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:51.031: INFO: Latency metrics for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:27:51.031: INFO: Logging node info for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:27:51.188: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-42-64.ap-northeast-1.compute.internal 45916c04-689e-47fa-b6fb-bdbad0784fcd 9817 0 2022-11-23 23:18:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-42-64.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-42-64.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-049ec10a59a1d20e4"} io.cilium.network.ipv4-cilium-host:172.20.51.104 io.cilium.network.ipv4-health-ip:172.20.34.63 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:22:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-049ec10a59a1d20e4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:56 +0000 UTC,LastTransitionTime:2022-11-23 23:18:56 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.64,},NodeAddress{Type:ExternalIP,Address:54.238.96.164,},NodeAddress{Type:Hostname,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-96-164.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25c6eb725883a2c2916889bd12be88,SystemUUID:ec25c6eb-7258-83a2-c291-6889bd12be88,BootID:8214bf1e-ee6e-4136-8902-31d30749be64,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:27:51.188: INFO: Logging kubelet events for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:27:51.347: INFO: Logging pods the kubelet thinks is on node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:27:51.507: INFO: coredns-5fcc7b6498-s4jjd started at 2022-11-23 23:19:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:51.507: INFO: Container coredns ready: true, restart count 0 Nov 23 23:27:51.507: INFO: pod-subpath-test-preprovisionedpv-rn5k started at 2022-11-23 23:27:48 +0000 UTC (2+2 container statuses recorded) Nov 23 23:27:51.507: INFO: Init container init-volume-preprovisionedpv-rn5k ready: true, restart count 0 Nov 23 23:27:51.507: INFO: Init container test-init-subpath-preprovisionedpv-rn5k ready: false, restart count 0 Nov 23 23:27:51.507: INFO: Container test-container-subpath-preprovisionedpv-rn5k ready: false, restart count 0 Nov 23 23:27:51.507: INFO: Container test-container-volume-preprovisionedpv-rn5k ready: false, restart count 0 Nov 23 23:27:51.507: INFO: hostexec-ip-172-20-42-64.ap-northeast-1.compute.internal-f75wd started at 2022-11-23 23:27:28 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:51.507: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:27:51.507: INFO: simpletest.rc-55r66 started at 2022-11-23 23:27:31 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:51.507: INFO: Container nginx ready: true, restart count 0 Nov 23 23:27:51.507: INFO: ebs-csi-node-fq5pw started at 2022-11-23 23:18:29 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:51.507: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:51.507: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:51.507: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:27:51.507: INFO: cilium-jjmjs started at 2022-11-23 23:18:29 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:51.507: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:27:51.507: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:27:51.507: INFO: hostpath-symlink-prep-provisioning-7216 started at 2022-11-23 23:27:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:51.507: INFO: Container init-volume-provisioning-7216 ready: false, restart count 0 Nov 23 23:27:51.507: INFO: simpletest.rc-8jbdw started at 2022-11-23 23:27:31 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:51.507: INFO: Container nginx ready: true, restart count 0 Nov 23 23:27:53.023: INFO: Latency metrics for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:27:53.023: INFO: Logging node info for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:27:53.191: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-200.ap-northeast-1.compute.internal 8f32a357-4fd8-4fcd-ab21-8b69dbd931d6 13757 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-200.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-6944":"csi-mock-csi-mock-volumes-6944","ebs.csi.aws.com":"i-0d4c764fea6a24b23"} io.cilium.network.ipv4-cilium-host:172.20.61.17 io.cilium.network.ipv4-health-ip:172.20.60.108 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:50 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:24:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0d4c764fea6a24b23,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:50 +0000 UTC,LastTransitionTime:2022-11-23 23:18:50 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:24:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.200,},NodeAddress{Type:ExternalIP,Address:3.113.30.221,},NodeAddress{Type:Hostname,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-113-30-221.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ef1c5e899b01a14fef22e61702fe5,SystemUUID:ec2ef1c5-e899-b01a-14fe-f22e61702fe5,BootID:021f6d58-0800-4a46-b65e-3513c48dff08,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:27:53.191: INFO: Logging kubelet events for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:27:53.349: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:27:53.511: INFO: ebs-csi-node-dl575 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:53.511: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:53.511: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:53.511: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:27:53.511: INFO: httpd started at 2022-11-23 23:27:41 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:53.511: INFO: Container httpd ready: true, restart count 0 Nov 23 23:27:53.511: INFO: logs-generator started at 2022-11-23 23:27:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:53.511: INFO: Container logs-generator ready: true, restart count 0 Nov 23 23:27:53.511: INFO: simpletest.rc-h2zcz started at 2022-11-23 23:27:31 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:53.511: INFO: Container nginx ready: true, restart count 0 Nov 23 23:27:53.511: INFO: csi-mockplugin-0 started at 2022-11-23 23:27:49 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:53.511: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:27:53.511: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:27:53.511: INFO: Container mock ready: true, restart count 0 Nov 23 23:27:53.511: INFO: simpletest.rc-rd5jf started at 2022-11-23 23:27:31 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:53.511: INFO: Container nginx ready: true, restart count 0 Nov 23 23:27:53.511: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:27:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:53.511: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:27:53.511: INFO: cilium-2lrb9 started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:53.511: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:27:53.511: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:27:53.511: INFO: coredns-autoscaler-6658b4bf85-gdgxf started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:53.511: INFO: Container autoscaler ready: true, restart count 0 Nov 23 23:27:53.511: INFO: coredns-5fcc7b6498-ll578 started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:53.511: INFO: Container coredns ready: true, restart count 0 Nov 23 23:27:54.065: INFO: Latency metrics for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:27:54.065: INFO: Logging node info for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:27:54.222: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-63-80.ap-northeast-1.compute.internal 1d4cdbf2-ca1f-431f-b562-a0bd0f489514 12148 0 2022-11-23 23:18:44 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-63-80.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-63-80.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-081735fbd2bce39f2"} io.cilium.network.ipv4-cilium-host:172.20.50.140 io.cilium.network.ipv4-health-ip:172.20.42.94 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:19:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:19:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:26:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-081735fbd2bce39f2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:19:12 +0000 UTC,LastTransitionTime:2022-11-23 23:19:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:26:24 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:26:24 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:26:24 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:26:24 +0000 UTC,LastTransitionTime:2022-11-23 23:19:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.63.80,},NodeAddress{Type:ExternalIP,Address:13.112.241.172,},NodeAddress{Type:Hostname,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-112-241-172.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29233394293c3c1bf3f4359c382b54,SystemUUID:ec292333-9429-3c3c-1bf3-f4359c382b54,BootID:05304de1-80b3-439c-9eeb-74d1d733f407,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:27:54.223: INFO: Logging kubelet events for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:27:54.381: INFO: Logging pods the kubelet thinks is on node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:27:54.542: INFO: simpletest.rc-6tjh2 started at 2022-11-23 23:27:31 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:54.542: INFO: Container nginx ready: true, restart count 0 Nov 23 23:27:54.542: INFO: simpletest.rc-gcs5f started at 2022-11-23 23:27:31 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:54.542: INFO: Container nginx ready: true, restart count 0 Nov 23 23:27:54.542: INFO: httpd started at 2022-11-23 23:27:41 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:54.542: INFO: Container httpd ready: true, restart count 0 Nov 23 23:27:54.542: INFO: run-log-test started at 2022-11-23 23:27:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:54.542: INFO: Container run-log-test ready: false, restart count 0 Nov 23 23:27:54.542: INFO: ebs-csi-node-hlgh2 started at 2022-11-23 23:18:44 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:54.542: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:54.542: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:54.542: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:27:54.542: INFO: startup-90ffdccd-1fcc-4f83-8ed9-18ea687b1842 started at 2022-11-23 23:27:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:54.542: INFO: Container busybox ready: true, restart count 0 Nov 23 23:27:54.542: INFO: simpletest.rc-gjw8k started at 2022-11-23 23:27:30 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:54.542: INFO: Container nginx ready: true, restart count 0 Nov 23 23:27:54.542: INFO: termination-message-containercd695921-6a58-4bab-954a-0a9c39752beb started at 2022-11-23 23:27:51 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:54.542: INFO: Container termination-message-container ready: false, restart count 0 Nov 23 23:27:54.542: INFO: hostexec-ip-172-20-63-80.ap-northeast-1.compute.internal-78mwj started at 2022-11-23 23:27:51 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:54.542: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:27:54.542: INFO: cilium-dgm5r started at 2022-11-23 23:18:44 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:54.542: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:27:54.542: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:27:54.542: INFO: pfpod started at 2022-11-23 23:27:37 +0000 UTC (0+2 container statuses recorded) Nov 23 23:27:54.542: INFO: Container portforwardtester ready: false, restart count 0 Nov 23 23:27:54.542: INFO: Container readiness ready: false, restart count 0 Nov 23 23:27:55.896: INFO: Latency metrics for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:27:55.896: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volumemode-1607" for this suite.
Filter through log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(block\svolmode\)\]\svolumes\sshould\sstore\sdata$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:159 Nov 23 23:41:14.840: Failed to create injector pod: timed out waiting for the condition /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186from junit_23.xml
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumes /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumes /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Nov 23 23:36:12.688: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename volume �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should store data /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:159 Nov 23 23:36:13.455: INFO: Creating resource for dynamic PV Nov 23 23:36:13.455: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass volume-4847-e2e-sc72scw �[1mSTEP�[0m: creating a claim �[1mSTEP�[0m: starting external-injector �[1mSTEP�[0m: Deleting pod external-injector in namespace volume-4847 Nov 23 23:41:14.688: INFO: Waiting for pod external-injector to disappear Nov 23 23:41:14.840: INFO: Pod external-injector no longer exists Nov 23 23:41:14.840: FAIL: Failed to create injector pod: timed out waiting for the condition Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumesTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186 +0x3ff k8s.io/kubernetes/test/e2e.RunE2ETests(0xc0002e9200) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc0002e9200) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc0002e9200, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 �[1mSTEP�[0m: cleaning the environment after external �[1mSTEP�[0m: Deleting pvc Nov 23 23:41:14.993: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comjgnxz" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (block volmode)] volumes /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "volume-4847". �[1mSTEP�[0m: Found 6 events. Nov 23 23:41:15.451: INFO: At 2022-11-23 23:36:13 +0000 UTC - event for ebs.csi.aws.comjgnxz: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 23:41:15.451: INFO: At 2022-11-23 23:36:14 +0000 UTC - event for ebs.csi.aws.comjgnxz: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 23:41:15.451: INFO: At 2022-11-23 23:36:14 +0000 UTC - event for ebs.csi.aws.comjgnxz: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } Provisioning: External provisioner is provisioning volume for claim "volume-4847/ebs.csi.aws.comjgnxz" Nov 23 23:41:15.451: INFO: At 2022-11-23 23:36:24 +0000 UTC - event for ebs.csi.aws.comjgnxz: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "volume-4847-e2e-sc72scw": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 23:41:15.451: INFO: At 2022-11-23 23:37:19 +0000 UTC - event for ebs.csi.aws.comjgnxz: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "volume-4847-e2e-sc72scw": rpc error: code = Internal desc = Could not create volume "pvc-e5e888ad-87ee-4007-88e1-1ad0de0c3ced": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Nov 23 23:41:15.451: INFO: At 2022-11-23 23:41:15 +0000 UTC - event for external-injector: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "external-injector" not found Nov 23 23:41:15.603: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 23:41:15.603: INFO: Nov 23 23:41:15.756: INFO: Logging node info for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:41:15.908: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-139.ap-northeast-1.compute.internal e52821e5-e82f-478b-be4f-6a60ba5816ce 39648 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-139.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-4383 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-35-139.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0af5c992eacc11f5f"} io.cilium.network.ipv4-cilium-host:172.20.62.73 io.cilium.network.ipv4-health-ip:172.20.63.125 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-11-23 23:27:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kubelet Update v1 2022-11-23 23:39:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0af5c992eacc11f5f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:51 +0000 UTC,LastTransitionTime:2022-11-23 23:18:51 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:39:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:39:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:39:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:39:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.139,},NodeAddress{Type:ExternalIP,Address:13.231.193.129,},NodeAddress{Type:Hostname,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-231-193-129.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec218b5e615d578e61275cbc46e5a775,SystemUUID:ec218b5e-615d-578e-6127-5cbc46e5a775,BootID:15a3520e-c72a-46fe-a3d6-be039aa283b7,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:41:15.909: INFO: Logging kubelet events for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:41:16.064: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:41:16.219: INFO: ebs-csi-node-vbdc7 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:41:16.219: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:41:16.219: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:41:16.219: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:41:16.219: INFO: hostexec-ip-172-20-35-139.ap-northeast-1.compute.internal-9vqgz started at 2022-11-23 23:40:56 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:16.219: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:41:16.219: INFO: webserver-pod started at 2022-11-23 23:39:24 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:16.220: INFO: Container agnhost ready: false, restart count 0 Nov 23 23:41:16.220: INFO: netserver-0 started at 2022-11-23 23:40:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:16.220: INFO: Container webserver ready: false, restart count 0 Nov 23 23:41:16.220: INFO: affinity-nodeport-lftgf started at 2022-11-23 23:41:08 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:16.220: INFO: Container affinity-nodeport ready: true, restart count 0 Nov 23 23:41:16.220: INFO: test-rolling-update-with-lb-7db9cf5ff8-z74lc started at 2022-11-23 23:38:48 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:16.220: INFO: Container agnhost ready: true, restart count 0 Nov 23 23:41:16.220: INFO: netserver-0 started at 2022-11-23 23:40:52 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:16.220: INFO: Container webserver ready: true, restart count 0 Nov 23 23:41:16.220: INFO: cilium-dw7qk started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:41:16.220: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:41:16.220: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:41:16.220: INFO: busybox-2b08b25b-fb2e-43ec-ac2f-4d23ffd0cfcf started at 2022-11-23 23:40:23 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:16.220: INFO: Container busybox ready: true, restart count 0 Nov 23 23:41:16.220: INFO: ss2-1 started at 2022-11-23 23:41:15 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:16.220: INFO: Container webserver ready: false, restart count 0 Nov 23 23:41:16.792: INFO: Latency metrics for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:41:16.792: INFO: Logging node info for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:41:16.944: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-241.ap-northeast-1.compute.internal 86ed4d71-4560-48af-9e88-d8d85c882f8b 38276 0 2022-11-23 23:16:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-241.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-027602f60f12db3f0"} io.cilium.network.ipv4-cilium-host:172.20.42.20 io.cilium.network.ipv4-health-ip:172.20.32.68 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 23:16:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-11-23 23:16:55 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-11-23 23:17:04 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:17:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 23:18:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:18:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-027602f60f12db3f0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892301824 0} {<nil>} 3801076Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787444224 0} {<nil>} 3698676Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:42 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:39:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:39:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:39:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:39:21 +0000 UTC,LastTransitionTime:2022-11-23 23:17:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.241,},NodeAddress{Type:ExternalIP,Address:54.238.84.104,},NodeAddress{Type:Hostname,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-84-104.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2adfc78fc274df02e3c085caa34395,SystemUUID:ec2adfc7-8fc2-74df-02e3-c085caa34395,BootID:e3997507-167a-4ec7-a6ec-7f97c5d886f5,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42904862,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41478108,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955301,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:41:16.945: INFO: Logging kubelet events for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:41:17.099: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:41:17.256: INFO: etcd-manager-main-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:17.256: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:41:17.257: INFO: kube-apiserver-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+2 container statuses recorded) Nov 23 23:41:17.257: INFO: Container healthcheck ready: true, restart count 0 Nov 23 23:41:17.257: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 23:41:17.257: INFO: ebs-csi-node-94jwj started at 2022-11-23 23:17:02 +0000 UTC (0+3 container statuses recorded) Nov 23 23:41:17.257: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:41:17.257: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:41:17.257: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:41:17.257: INFO: kops-controller-fznlr started at 2022-11-23 23:17:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:17.257: INFO: Container kops-controller ready: true, restart count 0 Nov 23 23:41:17.257: INFO: dns-controller-845449865f-zwg9n started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:17.257: INFO: Container dns-controller ready: true, restart count 0 Nov 23 23:41:17.257: INFO: kube-controller-manager-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:17.257: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 23:41:17.257: INFO: kube-scheduler-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:17.257: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 23:41:17.257: INFO: cilium-operator-587dd66f87-j2km7 started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:17.257: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 23:41:17.257: INFO: ebs-csi-controller-764b64d679-r8ttz started at 2022-11-23 23:17:03 +0000 UTC (0+5 container statuses recorded) Nov 23 23:41:17.257: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:41:17.257: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:41:17.257: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:41:17.257: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:41:17.257: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:41:17.257: INFO: etcd-manager-events-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:17.257: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:41:17.257: INFO: cilium-qvdj2 started at 2022-11-23 23:17:02 +0000 UTC (1+1 container statuses recorded) Nov 23 23:41:17.257: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:41:17.257: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 23:41:17.802: INFO: Latency metrics for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:41:17.802: INFO: Logging node info for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:41:17.956: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-42-64.ap-northeast-1.compute.internal 45916c04-689e-47fa-b6fb-bdbad0784fcd 41111 0 2022-11-23 23:18:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-42-64.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-42-64.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-049ec10a59a1d20e4"} io.cilium.network.ipv4-cilium-host:172.20.51.104 io.cilium.network.ipv4-health-ip:172.20.34.63 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:39:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-049ec10a59a1d20e4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:56 +0000 UTC,LastTransitionTime:2022-11-23 23:18:56 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:40:59 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:40:59 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:40:59 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:40:59 +0000 UTC,LastTransitionTime:2022-11-23 23:18:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.64,},NodeAddress{Type:ExternalIP,Address:54.238.96.164,},NodeAddress{Type:Hostname,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-96-164.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25c6eb725883a2c2916889bd12be88,SystemUUID:ec25c6eb-7258-83a2-c291-6889bd12be88,BootID:8214bf1e-ee6e-4136-8902-31d30749be64,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:41:17.957: INFO: Logging kubelet events for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:41:18.112: INFO: Logging pods the kubelet thinks is on node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:41:18.269: INFO: test-container-pod started at 2022-11-23 23:41:14 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:18.270: INFO: Container webserver ready: true, restart count 0 Nov 23 23:41:18.270: INFO: explicit-nonroot-uid started at 2022-11-23 23:41:17 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:18.270: INFO: Container explicit-nonroot-uid ready: false, restart count 0 Nov 23 23:41:18.270: INFO: liveness-c2299220-4849-440a-8618-bc3fcc381302 started at 2022-11-23 23:40:17 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:18.270: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:41:18.270: INFO: coredns-5fcc7b6498-s4jjd started at 2022-11-23 23:19:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:18.270: INFO: Container coredns ready: true, restart count 0 Nov 23 23:41:18.270: INFO: ebs-csi-node-fq5pw started at 2022-11-23 23:18:29 +0000 UTC (0+3 container statuses recorded) Nov 23 23:41:18.270: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:41:18.270: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:41:18.270: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:41:18.270: INFO: affinity-nodeport-h6hrw started at 2022-11-23 23:41:08 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:18.270: INFO: Container affinity-nodeport ready: true, restart count 0 Nov 23 23:41:18.270: INFO: concurrent-27820781-lcv4d started at 2022-11-23 23:41:00 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:18.270: INFO: Container c ready: false, restart count 0 Nov 23 23:41:18.270: INFO: ss2-0 started at 2022-11-23 23:40:44 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:18.270: INFO: Container webserver ready: true, restart count 0 Nov 23 23:41:18.270: INFO: netserver-1 started at 2022-11-23 23:40:52 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:18.270: INFO: Container webserver ready: true, restart count 0 Nov 23 23:41:18.270: INFO: netserver-1 started at 2022-11-23 23:40:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:18.270: INFO: Container webserver ready: true, restart count 0 Nov 23 23:41:18.270: INFO: downwardapi-volume-9f465899-23de-4275-9d13-147df7ae9d29 started at 2022-11-23 23:41:14 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:18.270: INFO: Container client-container ready: false, restart count 0 Nov 23 23:41:18.270: INFO: cilium-jjmjs started at 2022-11-23 23:18:29 +0000 UTC (1+1 container statuses recorded) Nov 23 23:41:18.270: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:41:18.270: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:41:20.829: INFO: Latency metrics for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:41:20.829: INFO: Logging node info for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:41:20.981: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-200.ap-northeast-1.compute.internal 8f32a357-4fd8-4fcd-ab21-8b69dbd931d6 40630 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-200.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-55-200.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d4c764fea6a24b23"} io.cilium.network.ipv4-cilium-host:172.20.61.17 io.cilium.network.ipv4-health-ip:172.20.60.108 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:50 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:40:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0d4c764fea6a24b23,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:50 +0000 UTC,LastTransitionTime:2022-11-23 23:18:50 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:40:33 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:40:33 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:40:33 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:40:33 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.200,},NodeAddress{Type:ExternalIP,Address:3.113.30.221,},NodeAddress{Type:Hostname,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-113-30-221.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ef1c5e899b01a14fef22e61702fe5,SystemUUID:ec2ef1c5-e899-b01a-14fe-f22e61702fe5,BootID:021f6d58-0800-4a46-b65e-3513c48dff08,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:41:20.982: INFO: Logging kubelet events for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:41:21.144: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:41:21.303: INFO: coredns-autoscaler-6658b4bf85-gdgxf started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:21.303: INFO: Container autoscaler ready: true, restart count 0 Nov 23 23:41:21.303: INFO: coredns-5fcc7b6498-ll578 started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:21.303: INFO: Container coredns ready: true, restart count 0 Nov 23 23:41:21.303: INFO: execpod-affinityrdmdz started at 2022-11-23 23:41:12 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:21.303: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:41:21.303: INFO: pause-pod-1 started at 2022-11-23 23:39:27 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:21.303: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:41:21.303: INFO: cilium-2lrb9 started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:41:21.303: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:41:21.303: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:41:21.303: INFO: netserver-2 started at 2022-11-23 23:40:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:21.303: INFO: Container webserver ready: true, restart count 0 Nov 23 23:41:21.303: INFO: probe-test-06890271-6377-4fa6-8441-9e79de654f13 started at 2022-11-23 23:41:16 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:21.303: INFO: Container probe-test-06890271-6377-4fa6-8441-9e79de654f13 ready: false, restart count 0 Nov 23 23:41:21.303: INFO: test-rolling-update-with-lb-7db9cf5ff8-2ld94 started at 2022-11-23 23:38:48 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:21.303: INFO: Container agnhost ready: true, restart count 0 Nov 23 23:41:21.303: INFO: ebs-csi-node-dl575 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:41:21.303: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:41:21.303: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:41:21.303: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:41:21.303: INFO: ss2-2 started at 2022-11-23 23:41:11 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:21.303: INFO: Container webserver ready: true, restart count 0 Nov 23 23:41:21.303: INFO: netserver-2 started at 2022-11-23 23:40:52 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:21.303: INFO: Container webserver ready: true, restart count 0 Nov 23 23:41:21.303: INFO: security-context-2fd372e9-9a64-46ca-9790-8ff36a1d899e started at 2022-11-23 23:41:19 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:21.303: INFO: Container test-container ready: false, restart count 0 Nov 23 23:41:21.303: INFO: affinity-nodeport-589h6 started at 2022-11-23 23:41:08 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:21.303: INFO: Container affinity-nodeport ready: true, restart count 0 Nov 23 23:41:21.836: INFO: Latency metrics for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:41:21.836: INFO: Logging node info for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:41:21.989: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-63-80.ap-northeast-1.compute.internal 1d4cdbf2-ca1f-431f-b562-a0bd0f489514 41271 0 2022-11-23 23:18:44 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-63-80.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-63-80.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-4567":"ip-172-20-63-80.ap-northeast-1.compute.internal","csi-mock-csi-mock-volumes-2255":"csi-mock-csi-mock-volumes-2255","ebs.csi.aws.com":"i-081735fbd2bce39f2"} io.cilium.network.ipv4-cilium-host:172.20.50.140 io.cilium.network.ipv4-health-ip:172.20.42.94 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:19:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:19:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 23:40:48 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 23:40:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-081735fbd2bce39f2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:19:12 +0000 UTC,LastTransitionTime:2022-11-23 23:19:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:41:05 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:41:05 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:41:05 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:41:05 +0000 UTC,LastTransitionTime:2022-11-23 23:19:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.63.80,},NodeAddress{Type:ExternalIP,Address:13.112.241.172,},NodeAddress{Type:Hostname,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-112-241-172.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29233394293c3c1bf3f4359c382b54,SystemUUID:ec292333-9429-3c3c-1bf3-f4359c382b54,BootID:05304de1-80b3-439c-9eeb-74d1d733f407,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-4567^419f4f86-6b88-11ed-afe2-928a4665fb09 kubernetes.io/csi/csi-hostpath-ephemeral-4567^48eba5dd-6b88-11ed-afe2-928a4665fb09 kubernetes.io/csi/csi-mock-csi-mock-volumes-2255^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-4567^419f4f86-6b88-11ed-afe2-928a4665fb09,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-2255^4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-4567^48eba5dd-6b88-11ed-afe2-928a4665fb09,DevicePath:,},},Config:nil,},} Nov 23 23:41:21.990: INFO: Logging kubelet events for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:41:22.144: INFO: Logging pods the kubelet thinks is on node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:41:22.321: INFO: ebs-csi-node-hlgh2 started at 2022-11-23 23:18:44 +0000 UTC (0+3 container statuses recorded) Nov 23 23:41:22.321: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:41:22.321: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:41:22.321: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:41:22.321: INFO: netserver-3 started at 2022-11-23 23:40:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:22.321: INFO: Container webserver ready: true, restart count 0 Nov 23 23:41:22.321: INFO: test-rolling-update-with-lb-7db9cf5ff8-fqv77 started at 2022-11-23 23:38:48 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:22.321: INFO: Container agnhost ready: true, restart count 0 Nov 23 23:41:22.321: INFO: csi-hostpathplugin-0 started at 2022-11-23 23:40:43 +0000 UTC (0+7 container statuses recorded) Nov 23 23:41:22.321: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:41:22.321: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:41:22.321: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:41:22.321: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 23 23:41:22.321: INFO: Container hostpath ready: true, restart count 0 Nov 23 23:41:22.321: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:41:22.321: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:41:22.321: INFO: csi-mockplugin-resizer-0 started at 2022-11-23 23:40:45 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:22.321: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:41:22.321: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:40:45 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:22.321: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:41:22.321: INFO: inline-volume-tester-bl9n6 started at 2022-11-23 23:40:48 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:22.321: INFO: Container csi-volume-tester ready: true, restart count 0 Nov 23 23:41:22.321: INFO: netserver-3 started at 2022-11-23 23:40:52 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:22.321: INFO: Container webserver ready: true, restart count 0 Nov 23 23:41:22.321: INFO: inline-volume-tester2-vncgr started at 2022-11-23 23:41:01 +0000 UTC (0+1 container statuses recorded) Nov 23 23:41:22.321: INFO: Container csi-volume-tester ready: true, restart count 0 Nov 23 23:41:22.321: INFO: cilium-dgm5r started at 2022-11-23 23:18:44 +0000 UTC (1+1 container statuses recorded) Nov 23 23:41:22.321: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:41:22.321: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:41:22.321: INFO: csi-mockplugin-0 started at 2022-11-23 23:40:44 +0000 UTC (0+3 container statuses recorded) Nov 23 23:41:22.321: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:41:22.321: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:41:22.321: INFO: Container mock ready: true, restart count 0 Nov 23 23:41:22.321: INFO: pod-secrets-e3ab06b5-9c76-496d-83c1-5b2a6122e3d1 started at 2022-11-23 23:39:59 +0000 UTC (0+3 container statuses recorded) Nov 23 23:41:22.321: INFO: Container creates-volume-test ready: true, restart count 0 Nov 23 23:41:22.321: INFO: Container dels-volume-test ready: true, restart count 0 Nov 23 23:41:22.321: INFO: Container upds-volume-test ready: true, restart count 0 Nov 23 23:41:22.925: INFO: Latency metrics for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:41:22.925: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volume-4847" for this suite.
Filter through log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\(allowExpansion\)\]\svolume\-expand\sVerify\sif\soffline\sPVC\sexpansion\sworks$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:174 Nov 23 23:28:55.510: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc004e28bc0>: { s: "pod \"pod-801e9bb0-73ca-43a9-afc6-2b3500da9b18\" is not Running: timed out waiting for the condition", } pod "pod-801e9bb0-73ca-43a9-afc6-2b3500da9b18" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:192from junit_24.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Nov 23 23:23:53.602: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename volume-expand �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] Verify if offline PVC expansion works /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:174 Nov 23 23:23:54.396: INFO: Creating resource for dynamic PV Nov 23 23:23:54.396: INFO: Using claimSize:1Gi, test suite supported size:{ 1Gi}, driver(ebs.csi.aws.com) supported size:{ 1Gi} �[1mSTEP�[0m: creating a StorageClass volume-expand-8815-e2e-sclx6qf �[1mSTEP�[0m: creating a claim Nov 23 23:23:54.558: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating a pod with dynamically provisioned volume Nov 23 23:28:55.510: FAIL: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc004e28bc0>: { s: "pod \"pod-801e9bb0-73ca-43a9-afc6-2b3500da9b18\" is not Running: timed out waiting for the condition", } pod "pod-801e9bb0-73ca-43a9-afc6-2b3500da9b18" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func4() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:192 +0x34d k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000c79080) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000c79080) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000c79080, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 Nov 23 23:28:55.511: INFO: Deleting pod "pod-801e9bb0-73ca-43a9-afc6-2b3500da9b18" in namespace "volume-expand-8815" Nov 23 23:28:55.675: INFO: Wait up to 5m0s for pod "pod-801e9bb0-73ca-43a9-afc6-2b3500da9b18" to be fully deleted �[1mSTEP�[0m: Deleting pod Nov 23 23:28:55.833: INFO: Deleting pod "pod-801e9bb0-73ca-43a9-afc6-2b3500da9b18" in namespace "volume-expand-8815" �[1mSTEP�[0m: Deleting pvc Nov 23 23:28:56.149: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.compcdxb" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "volume-expand-8815". �[1mSTEP�[0m: Found 6 events. Nov 23 23:28:56.625: INFO: At 2022-11-23 23:23:54 +0000 UTC - event for ebs.csi.aws.compcdxb: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 23:28:56.625: INFO: At 2022-11-23 23:23:54 +0000 UTC - event for ebs.csi.aws.compcdxb: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 23:28:56.625: INFO: At 2022-11-23 23:23:54 +0000 UTC - event for ebs.csi.aws.compcdxb: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } Provisioning: External provisioner is provisioning volume for claim "volume-expand-8815/ebs.csi.aws.compcdxb" Nov 23 23:28:56.625: INFO: At 2022-11-23 23:24:04 +0000 UTC - event for ebs.csi.aws.compcdxb: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-8815-e2e-sclx6qf": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 23:28:56.625: INFO: At 2022-11-23 23:27:21 +0000 UTC - event for ebs.csi.aws.compcdxb: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-8815-e2e-sclx6qf": rpc error: code = Internal desc = Could not create volume "pvc-cf2187cc-82a2-4552-90d7-05cc20c03b8a": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Nov 23 23:28:56.625: INFO: At 2022-11-23 23:28:55 +0000 UTC - event for pod-801e9bb0-73ca-43a9-afc6-2b3500da9b18: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "pod-801e9bb0-73ca-43a9-afc6-2b3500da9b18" not found Nov 23 23:28:56.782: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 23:28:56.783: INFO: Nov 23 23:28:56.941: INFO: Logging node info for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:28:57.100: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-139.ap-northeast-1.compute.internal e52821e5-e82f-478b-be4f-6a60ba5816ce 14113 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-139.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-4383 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-35-139.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-4950":"csi-mock-csi-mock-volumes-4950","ebs.csi.aws.com":"i-0af5c992eacc11f5f"} io.cilium.network.ipv4-cilium-host:172.20.62.73 io.cilium.network.ipv4-health-ip:172.20.63.125 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-11-23 23:27:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:27:56 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 23:28:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0af5c992eacc11f5f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:51 +0000 UTC,LastTransitionTime:2022-11-23 23:18:51 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:28:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:28:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:28:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:28:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.139,},NodeAddress{Type:ExternalIP,Address:13.231.193.129,},NodeAddress{Type:Hostname,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-231-193-129.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec218b5e615d578e61275cbc46e5a775,SystemUUID:ec218b5e-615d-578e-6127-5cbc46e5a775,BootID:15a3520e-c72a-46fe-a3d6-be039aa283b7,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-4950^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-4950^4,DevicePath:,},},Config:nil,},} Nov 23 23:28:57.100: INFO: Logging kubelet events for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:28:57.260: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:28:57.427: INFO: csi-mockplugin-0 started at 2022-11-23 23:27:48 +0000 UTC (0+3 container statuses recorded) Nov 23 23:28:57.428: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:28:57.428: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:28:57.428: INFO: Container mock ready: true, restart count 0 Nov 23 23:28:57.428: INFO: ss2-0 started at 2022-11-23 23:28:43 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:57.428: INFO: Container webserver ready: true, restart count 0 Nov 23 23:28:57.428: INFO: rs-km877 started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:57.428: INFO: Container donothing ready: false, restart count 0 Nov 23 23:28:57.428: INFO: rs-5lh9m started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:57.428: INFO: Container donothing ready: false, restart count 0 Nov 23 23:28:57.428: INFO: all-succeed-6hqpw started at 2022-11-23 23:28:45 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:57.428: INFO: Container c ready: false, restart count 0 Nov 23 23:28:57.428: INFO: all-succeed-kxmrg started at 2022-11-23 23:28:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:57.428: INFO: Container c ready: false, restart count 0 Nov 23 23:28:57.428: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:27:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:57.428: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:28:57.428: INFO: pvc-volume-tester-cjrdx started at 2022-11-23 23:27:55 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:57.428: INFO: Container volume-tester ready: true, restart count 0 Nov 23 23:28:57.428: INFO: frontend-74dc45c94b-b49rc started at 2022-11-23 23:28:55 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:57.428: INFO: Container guestbook-frontend ready: false, restart count 0 Nov 23 23:28:57.428: INFO: cilium-dw7qk started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:28:57.428: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:28:57.428: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:28:57.428: INFO: rs-nk4kz started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:57.428: INFO: Container donothing ready: false, restart count 0 Nov 23 23:28:57.428: INFO: ebs-csi-node-vbdc7 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:28:57.428: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:28:57.428: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:28:57.428: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:28:58.237: INFO: Latency metrics for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:28:58.238: INFO: Logging node info for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:28:58.396: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-241.ap-northeast-1.compute.internal 86ed4d71-4560-48af-9e88-d8d85c882f8b 6905 0 2022-11-23 23:16:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-241.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-027602f60f12db3f0"} io.cilium.network.ipv4-cilium-host:172.20.42.20 io.cilium.network.ipv4-health-ip:172.20.32.68 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 23:16:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-11-23 23:16:55 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-11-23 23:17:04 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:17:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 23:18:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:18:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-027602f60f12db3f0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892301824 0} {<nil>} 3801076Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787444224 0} {<nil>} 3698676Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:42 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:17:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.241,},NodeAddress{Type:ExternalIP,Address:54.238.84.104,},NodeAddress{Type:Hostname,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-84-104.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2adfc78fc274df02e3c085caa34395,SystemUUID:ec2adfc7-8fc2-74df-02e3-c085caa34395,BootID:e3997507-167a-4ec7-a6ec-7f97c5d886f5,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42904862,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41478108,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955301,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:28:58.396: INFO: Logging kubelet events for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:28:58.557: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:28:58.719: INFO: kops-controller-fznlr started at 2022-11-23 23:17:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:58.719: INFO: Container kops-controller ready: true, restart count 0 Nov 23 23:28:58.720: INFO: dns-controller-845449865f-zwg9n started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:58.720: INFO: Container dns-controller ready: true, restart count 0 Nov 23 23:28:58.720: INFO: kube-controller-manager-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:58.720: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 23:28:58.720: INFO: kube-scheduler-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:58.720: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 23:28:58.720: INFO: etcd-manager-main-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:58.720: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:28:58.720: INFO: kube-apiserver-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+2 container statuses recorded) Nov 23 23:28:58.720: INFO: Container healthcheck ready: true, restart count 0 Nov 23 23:28:58.720: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 23:28:58.720: INFO: ebs-csi-node-94jwj started at 2022-11-23 23:17:02 +0000 UTC (0+3 container statuses recorded) Nov 23 23:28:58.720: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:28:58.720: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:28:58.720: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:28:58.720: INFO: etcd-manager-events-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:58.720: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:28:58.720: INFO: cilium-qvdj2 started at 2022-11-23 23:17:02 +0000 UTC (1+1 container statuses recorded) Nov 23 23:28:58.720: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:28:58.720: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 23:28:58.720: INFO: cilium-operator-587dd66f87-j2km7 started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:58.720: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 23:28:58.720: INFO: ebs-csi-controller-764b64d679-r8ttz started at 2022-11-23 23:17:03 +0000 UTC (0+5 container statuses recorded) Nov 23 23:28:58.720: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:28:58.720: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:28:58.720: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:28:58.720: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:28:58.720: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:28:59.328: INFO: Latency metrics for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:28:59.328: INFO: Logging node info for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:28:59.486: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-42-64.ap-northeast-1.compute.internal 45916c04-689e-47fa-b6fb-bdbad0784fcd 9817 0 2022-11-23 23:18:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-42-64.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-42-64.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-049ec10a59a1d20e4"} io.cilium.network.ipv4-cilium-host:172.20.51.104 io.cilium.network.ipv4-health-ip:172.20.34.63 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:22:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-049ec10a59a1d20e4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:56 +0000 UTC,LastTransitionTime:2022-11-23 23:18:56 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.64,},NodeAddress{Type:ExternalIP,Address:54.238.96.164,},NodeAddress{Type:Hostname,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-96-164.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25c6eb725883a2c2916889bd12be88,SystemUUID:ec25c6eb-7258-83a2-c291-6889bd12be88,BootID:8214bf1e-ee6e-4136-8902-31d30749be64,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:28:59.487: INFO: Logging kubelet events for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:28:59.647: INFO: Logging pods the kubelet thinks is on node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:28:59.813: INFO: inline-volume-c2zwr started at 2022-11-23 23:27:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:59.813: INFO: Container volume-tester ready: false, restart count 0 Nov 23 23:28:59.813: INFO: inline-volume-xhkps started at 2022-11-23 23:28:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:59.813: INFO: Container volume-tester ready: false, restart count 0 Nov 23 23:28:59.813: INFO: agnhost-replica-6987b74bb4-hf28s started at 2022-11-23 23:28:56 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:59.813: INFO: Container replica ready: true, restart count 0 Nov 23 23:28:59.813: INFO: cilium-jjmjs started at 2022-11-23 23:18:29 +0000 UTC (1+1 container statuses recorded) Nov 23 23:28:59.813: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:28:59.813: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:28:59.813: INFO: pod-subpath-test-configmap-bnr4 started at 2022-11-23 23:28:38 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:59.813: INFO: Container test-container-subpath-configmap-bnr4 ready: true, restart count 0 Nov 23 23:28:59.813: INFO: startup-73da1a16-b444-4ab8-955f-74bbd9876a25 started at 2022-11-23 23:28:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:59.813: INFO: Container busybox ready: false, restart count 0 Nov 23 23:28:59.813: INFO: pod-service-account-mountsa started at 2022-11-23 23:28:56 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:59.814: INFO: Container token-test ready: true, restart count 0 Nov 23 23:28:59.814: INFO: coredns-5fcc7b6498-s4jjd started at 2022-11-23 23:19:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:59.814: INFO: Container coredns ready: true, restart count 0 Nov 23 23:28:59.814: INFO: rs-z5brl started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:59.814: INFO: Container donothing ready: true, restart count 0 Nov 23 23:28:59.814: INFO: ebs-csi-node-fq5pw started at 2022-11-23 23:18:29 +0000 UTC (0+3 container statuses recorded) Nov 23 23:28:59.814: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:28:59.814: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:28:59.814: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:28:59.814: INFO: rs-zkqnm started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:59.814: INFO: Container donothing ready: true, restart count 0 Nov 23 23:28:59.814: INFO: frontend-74dc45c94b-mxxhx started at 2022-11-23 23:28:55 +0000 UTC (0+1 container statuses recorded) Nov 23 23:28:59.814: INFO: Container guestbook-frontend ready: true, restart count 0 Nov 23 23:29:00.337: INFO: Latency metrics for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:29:00.337: INFO: Logging node info for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:29:00.496: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-200.ap-northeast-1.compute.internal 8f32a357-4fd8-4fcd-ab21-8b69dbd931d6 13785 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-200.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-6944":"csi-mock-csi-mock-volumes-6944","ebs.csi.aws.com":"i-0d4c764fea6a24b23"} io.cilium.network.ipv4-cilium-host:172.20.61.17 io.cilium.network.ipv4-health-ip:172.20.60.108 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:50 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:24:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0d4c764fea6a24b23,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:50 +0000 UTC,LastTransitionTime:2022-11-23 23:18:50 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.200,},NodeAddress{Type:ExternalIP,Address:3.113.30.221,},NodeAddress{Type:Hostname,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-113-30-221.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ef1c5e899b01a14fef22e61702fe5,SystemUUID:ec2ef1c5-e899-b01a-14fe-f22e61702fe5,BootID:021f6d58-0800-4a46-b65e-3513c48dff08,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:29:00.496: INFO: Logging kubelet events for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:29:00.656: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:29:00.820: INFO: ebs-csi-node-dl575 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:29:00.820: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:29:00.820: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:29:00.820: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:29:00.820: INFO: pvc-volume-tester-wrf6r started at 2022-11-23 23:27:55 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:00.820: INFO: Container volume-tester ready: true, restart count 0 Nov 23 23:29:00.820: INFO: rs-mg6vl started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:00.820: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:00.820: INFO: cilium-2lrb9 started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:29:00.820: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:29:00.820: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:29:00.820: INFO: coredns-autoscaler-6658b4bf85-gdgxf started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:00.820: INFO: Container autoscaler ready: true, restart count 0 Nov 23 23:29:00.820: INFO: coredns-5fcc7b6498-ll578 started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:00.820: INFO: Container coredns ready: true, restart count 0 Nov 23 23:29:00.820: INFO: rs-dthbz started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:00.820: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:00.820: INFO: csi-mockplugin-0 started at 2022-11-23 23:27:49 +0000 UTC (0+3 container statuses recorded) Nov 23 23:29:00.820: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:29:00.820: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:29:00.820: INFO: Container mock ready: true, restart count 0 Nov 23 23:29:00.820: INFO: pod-service-account-nomountsa started at 2022-11-23 23:28:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:00.820: INFO: Container token-test ready: true, restart count 0 Nov 23 23:29:00.820: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:27:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:00.820: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:29:00.820: INFO: pod-0 started at 2022-11-23 23:28:01 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:00.820: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:00.820: INFO: pod-2 started at 2022-11-23 23:28:01 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:00.820: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:00.820: INFO: startup-script started at 2022-11-23 23:28:04 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:00.820: INFO: Container startup-script ready: true, restart count 0 Nov 23 23:29:01.388: INFO: Latency metrics for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:29:01.388: INFO: Logging node info for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:29:01.546: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-63-80.ap-northeast-1.compute.internal 1d4cdbf2-ca1f-431f-b562-a0bd0f489514 15090 0 2022-11-23 23:18:44 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-63-80.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-63-80.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-081735fbd2bce39f2"} io.cilium.network.ipv4-cilium-host:172.20.50.140 io.cilium.network.ipv4-health-ip:172.20.42.94 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:19:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:19:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:26:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-081735fbd2bce39f2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:19:12 +0000 UTC,LastTransitionTime:2022-11-23 23:19:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:28:44 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:28:44 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:28:44 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:28:44 +0000 UTC,LastTransitionTime:2022-11-23 23:19:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.63.80,},NodeAddress{Type:ExternalIP,Address:13.112.241.172,},NodeAddress{Type:Hostname,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-112-241-172.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29233394293c3c1bf3f4359c382b54,SystemUUID:ec292333-9429-3c3c-1bf3-f4359c382b54,BootID:05304de1-80b3-439c-9eeb-74d1d733f407,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:29:01.547: INFO: Logging kubelet events for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:29:01.708: INFO: Logging pods the kubelet thinks is on node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:29:01.871: INFO: alpine-nnp-false-c013d00a-fa87-43ee-897a-7a5a250c6770 started at <nil> (0+0 container statuses recorded) Nov 23 23:29:01.871: INFO: pod-logs-websocket-efa8f17c-4ebe-4b90-8a38-1592a7ce896c started at 2022-11-23 23:28:37 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:01.871: INFO: Container main ready: true, restart count 0 Nov 23 23:29:01.871: INFO: rs-fv8pd started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:01.871: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:01.871: INFO: frontend-74dc45c94b-wzt7q started at 2022-11-23 23:28:55 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:01.871: INFO: Container guestbook-frontend ready: false, restart count 0 Nov 23 23:29:01.871: INFO: pod-service-account-defaultsa started at 2022-11-23 23:28:56 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:01.871: INFO: Container token-test ready: false, restart count 0 Nov 23 23:29:01.871: INFO: rs-scn49 started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:01.871: INFO: Container donothing ready: false, restart count 0 Nov 23 23:29:01.871: INFO: suspend-false-to-true-4wtmf started at 2022-11-23 23:28:23 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:01.871: INFO: Container c ready: true, restart count 0 Nov 23 23:29:01.871: INFO: cilium-dgm5r started at 2022-11-23 23:18:44 +0000 UTC (1+1 container statuses recorded) Nov 23 23:29:01.871: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:29:01.871: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:29:01.871: INFO: boom-server started at 2022-11-23 23:27:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:01.871: INFO: Container boom-server ready: true, restart count 0 Nov 23 23:29:01.871: INFO: pod-1 started at 2022-11-23 23:28:01 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:01.871: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:01.871: INFO: agnhost-primary-58467dcb76-jz5jh started at 2022-11-23 23:28:55 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:01.871: INFO: Container primary ready: false, restart count 0 Nov 23 23:29:01.871: INFO: rs-wrj5z started at 2022-11-23 23:28:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:01.871: INFO: Container donothing ready: true, restart count 0 Nov 23 23:29:01.871: INFO: agnhost-replica-6987b74bb4-r5zbl started at 2022-11-23 23:28:56 +0000 UTC (0+1 container statuses recorded) Nov 23 23:29:01.871: INFO: Container replica ready: false, restart count 0 Nov 23 23:29:01.871: INFO: ebs-csi-node-hlgh2 started at 2022-11-23 23:18:44 +0000 UTC (0+3 container statuses recorded) Nov 23 23:29:01.871: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:29:01.871: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:29:01.871: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:29:01.871: INFO: pod-service-account-defaultsa-mountspec started at <nil> (0+0 container statuses recorded) Nov 23 23:29:02.727: INFO: Latency metrics for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:29:02.727: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volume-expand-8815" for this suite.
Find pod-801e9bb0-73ca-43a9-afc6-2b3500da9b18 mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\(allowExpansion\)\]\svolume\-expand\sshould\sresize\svolume\swhen\sPVC\sis\sedited\swhile\spod\sis\susing\sit$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:246 Nov 23 23:32:29.870: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc0044a4af0>: { s: "pod \"pod-bdad766a-25d1-423f-a111-7df3bb88031a\" is not Running: timed out waiting for the condition", } pod "pod-bdad766a-25d1-423f-a111-7df3bb88031a" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:268from junit_22.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Nov 23 23:27:28.035: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename volume-expand �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should resize volume when PVC is edited while pod is using it /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:246 Nov 23 23:27:28.794: INFO: Creating resource for dynamic PV Nov 23 23:27:28.794: INFO: Using claimSize:1Gi, test suite supported size:{ 1Gi}, driver(ebs.csi.aws.com) supported size:{ 1Gi} �[1mSTEP�[0m: creating a StorageClass volume-expand-5399-e2e-scdkpj6 �[1mSTEP�[0m: creating a claim Nov 23 23:27:28.947: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating a pod with dynamically provisioned volume Nov 23 23:32:29.870: FAIL: While creating pods for resizing Unexpected error: <*errors.errorString | 0xc0044a4af0>: { s: "pod \"pod-bdad766a-25d1-423f-a111-7df3bb88031a\" is not Running: timed out waiting for the condition", } pod "pod-bdad766a-25d1-423f-a111-7df3bb88031a" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func5() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:268 +0x3d0 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000998180) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000998180) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000998180, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 Nov 23 23:32:29.870: INFO: Deleting pod "pod-bdad766a-25d1-423f-a111-7df3bb88031a" in namespace "volume-expand-5399" Nov 23 23:32:30.026: INFO: Wait up to 5m0s for pod "pod-bdad766a-25d1-423f-a111-7df3bb88031a" to be fully deleted �[1mSTEP�[0m: Deleting pod Nov 23 23:32:30.177: INFO: Deleting pod "pod-bdad766a-25d1-423f-a111-7df3bb88031a" in namespace "volume-expand-5399" �[1mSTEP�[0m: Deleting pvc Nov 23 23:32:30.482: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comgldw6" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "volume-expand-5399". �[1mSTEP�[0m: Found 5 events. Nov 23 23:32:30.939: INFO: At 2022-11-23 23:27:29 +0000 UTC - event for ebs.csi.aws.comgldw6: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 23:32:30.939: INFO: At 2022-11-23 23:27:29 +0000 UTC - event for ebs.csi.aws.comgldw6: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 23:32:30.939: INFO: At 2022-11-23 23:27:29 +0000 UTC - event for ebs.csi.aws.comgldw6: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } Provisioning: External provisioner is provisioning volume for claim "volume-expand-5399/ebs.csi.aws.comgldw6" Nov 23 23:32:30.939: INFO: At 2022-11-23 23:27:39 +0000 UTC - event for ebs.csi.aws.comgldw6: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-5399-e2e-scdkpj6": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 23:32:30.939: INFO: At 2022-11-23 23:32:30 +0000 UTC - event for pod-bdad766a-25d1-423f-a111-7df3bb88031a: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "pod-bdad766a-25d1-423f-a111-7df3bb88031a" not found Nov 23 23:32:31.090: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 23:32:31.090: INFO: Nov 23 23:32:31.243: INFO: Logging node info for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:32:31.394: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-139.ap-northeast-1.compute.internal e52821e5-e82f-478b-be4f-6a60ba5816ce 21725 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-139.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-4383 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-35-139.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0af5c992eacc11f5f"} io.cilium.network.ipv4-cilium-host:172.20.62.73 io.cilium.network.ipv4-health-ip:172.20.63.125 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-11-23 23:27:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kubelet Update v1 2022-11-23 23:31:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0af5c992eacc11f5f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:51 +0000 UTC,LastTransitionTime:2022-11-23 23:18:51 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.139,},NodeAddress{Type:ExternalIP,Address:13.231.193.129,},NodeAddress{Type:Hostname,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-231-193-129.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec218b5e615d578e61275cbc46e5a775,SystemUUID:ec218b5e-615d-578e-6127-5cbc46e5a775,BootID:15a3520e-c72a-46fe-a3d6-be039aa283b7,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:32:31.395: INFO: Logging kubelet events for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:32:31.551: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:32:31.708: INFO: netserver-0 started at 2022-11-23 23:32:16 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:31.708: INFO: Container webserver ready: false, restart count 0 Nov 23 23:32:31.708: INFO: cilium-dw7qk started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:32:31.708: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:32:31.708: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:32:31.708: INFO: var-expansion-650f6699-ec38-4b41-9e4f-71fd28ee49a2 started at 2022-11-23 23:32:30 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:31.708: INFO: Container dapi-container ready: false, restart count 0 Nov 23 23:32:31.708: INFO: ss2-0 started at 2022-11-23 23:32:14 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:31.708: INFO: Container webserver ready: true, restart count 0 Nov 23 23:32:31.708: INFO: ebs-csi-node-vbdc7 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:31.708: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:31.708: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:31.708: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:31.708: INFO: dns-test-ff5de24a-ed0e-4484-a1f7-66cf4cfed751 started at 2022-11-23 23:32:29 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:31.708: INFO: Container jessie-querier ready: true, restart count 0 Nov 23 23:32:31.708: INFO: Container querier ready: true, restart count 0 Nov 23 23:32:31.708: INFO: Container webserver ready: true, restart count 0 Nov 23 23:32:34.657: INFO: Latency metrics for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:32:34.657: INFO: Logging node info for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:32:34.809: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-241.ap-northeast-1.compute.internal 86ed4d71-4560-48af-9e88-d8d85c882f8b 16487 0 2022-11-23 23:16:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-241.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-027602f60f12db3f0"} io.cilium.network.ipv4-cilium-host:172.20.42.20 io.cilium.network.ipv4-health-ip:172.20.32.68 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 23:16:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-11-23 23:16:55 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-11-23 23:17:04 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:17:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 23:18:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:18:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-027602f60f12db3f0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892301824 0} {<nil>} 3801076Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787444224 0} {<nil>} 3698676Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:42 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:17:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.241,},NodeAddress{Type:ExternalIP,Address:54.238.84.104,},NodeAddress{Type:Hostname,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-84-104.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2adfc78fc274df02e3c085caa34395,SystemUUID:ec2adfc7-8fc2-74df-02e3-c085caa34395,BootID:e3997507-167a-4ec7-a6ec-7f97c5d886f5,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42904862,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41478108,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955301,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:32:34.809: INFO: Logging kubelet events for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:32:34.964: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:32:35.273: INFO: kops-controller-fznlr started at 2022-11-23 23:17:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:35.273: INFO: Container kops-controller ready: true, restart count 0 Nov 23 23:32:35.273: INFO: dns-controller-845449865f-zwg9n started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:35.273: INFO: Container dns-controller ready: true, restart count 0 Nov 23 23:32:35.273: INFO: kube-controller-manager-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:35.273: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 23:32:35.273: INFO: kube-scheduler-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:35.273: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 23:32:35.273: INFO: etcd-manager-main-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:35.273: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:32:35.273: INFO: kube-apiserver-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+2 container statuses recorded) Nov 23 23:32:35.273: INFO: Container healthcheck ready: true, restart count 0 Nov 23 23:32:35.273: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 23:32:35.273: INFO: ebs-csi-node-94jwj started at 2022-11-23 23:17:02 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:35.273: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:35.273: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:35.273: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:35.273: INFO: etcd-manager-events-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:35.273: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:32:35.273: INFO: cilium-qvdj2 started at 2022-11-23 23:17:02 +0000 UTC (1+1 container statuses recorded) Nov 23 23:32:35.273: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:32:35.273: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 23:32:35.273: INFO: cilium-operator-587dd66f87-j2km7 started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:35.273: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 23:32:35.273: INFO: ebs-csi-controller-764b64d679-r8ttz started at 2022-11-23 23:17:03 +0000 UTC (0+5 container statuses recorded) Nov 23 23:32:35.273: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:32:35.273: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:32:35.273: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:32:35.273: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:35.273: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:35.816: INFO: Latency metrics for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:32:35.817: INFO: Logging node info for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:32:36.013: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-42-64.ap-northeast-1.compute.internal 45916c04-689e-47fa-b6fb-bdbad0784fcd 24098 0 2022-11-23 23:18:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-42-64.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-42-64.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-049ec10a59a1d20e4"} io.cilium.network.ipv4-cilium-host:172.20.51.104 io.cilium.network.ipv4-health-ip:172.20.34.63 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:32:09 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-049ec10a59a1d20e4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:56 +0000 UTC,LastTransitionTime:2022-11-23 23:18:56 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:32:09 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:32:09 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:32:09 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:32:09 +0000 UTC,LastTransitionTime:2022-11-23 23:18:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.64,},NodeAddress{Type:ExternalIP,Address:54.238.96.164,},NodeAddress{Type:Hostname,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-96-164.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25c6eb725883a2c2916889bd12be88,SystemUUID:ec25c6eb-7258-83a2-c291-6889bd12be88,BootID:8214bf1e-ee6e-4136-8902-31d30749be64,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:32:36.014: INFO: Logging kubelet events for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:32:36.167: INFO: Logging pods the kubelet thinks is on node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:32:36.323: INFO: ebs-csi-node-fq5pw started at 2022-11-23 23:18:29 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:36.323: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:36.323: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:36.323: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:36.323: INFO: netserver-1 started at 2022-11-23 23:32:16 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:36.323: INFO: Container webserver ready: false, restart count 0 Nov 23 23:32:36.323: INFO: netserver-1 started at 2022-11-23 23:32:33 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:36.323: INFO: Container webserver ready: false, restart count 0 Nov 23 23:32:36.323: INFO: cilium-jjmjs started at 2022-11-23 23:18:29 +0000 UTC (1+1 container statuses recorded) Nov 23 23:32:36.323: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:32:36.323: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:32:36.323: INFO: coredns-5fcc7b6498-s4jjd started at 2022-11-23 23:19:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:36.323: INFO: Container coredns ready: true, restart count 0 Nov 23 23:32:36.323: INFO: pod-ready started at 2022-11-23 23:32:24 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:36.323: INFO: Container pod-readiness-gate ready: true, restart count 0 Nov 23 23:32:36.826: INFO: Latency metrics for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:32:36.826: INFO: Logging node info for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:32:36.978: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-200.ap-northeast-1.compute.internal 8f32a357-4fd8-4fcd-ab21-8b69dbd931d6 23091 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-200.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9079":"csi-mock-csi-mock-volumes-9079","ebs.csi.aws.com":"i-0d4c764fea6a24b23"} io.cilium.network.ipv4-cilium-host:172.20.61.17 io.cilium.network.ipv4-health-ip:172.20.60.108 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:50 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:24:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0d4c764fea6a24b23,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:50 +0000 UTC,LastTransitionTime:2022-11-23 23:18:50 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.200,},NodeAddress{Type:ExternalIP,Address:3.113.30.221,},NodeAddress{Type:Hostname,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-113-30-221.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ef1c5e899b01a14fef22e61702fe5,SystemUUID:ec2ef1c5-e899-b01a-14fe-f22e61702fe5,BootID:021f6d58-0800-4a46-b65e-3513c48dff08,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:32:36.979: INFO: Logging kubelet events for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:32:37.134: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:32:37.290: INFO: netserver-2 started at 2022-11-23 23:32:16 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:37.290: INFO: Container webserver ready: false, restart count 0 Nov 23 23:32:37.290: INFO: pvc-volume-tester-qtr6f started at 2022-11-23 23:32:04 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:37.290: INFO: Container volume-tester ready: true, restart count 0 Nov 23 23:32:37.290: INFO: coredns-autoscaler-6658b4bf85-gdgxf started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:37.290: INFO: Container autoscaler ready: true, restart count 0 Nov 23 23:32:37.290: INFO: coredns-5fcc7b6498-ll578 started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:37.290: INFO: Container coredns ready: true, restart count 0 Nov 23 23:32:37.290: INFO: liveness-c01c2deb-84cd-4e87-b47a-aab4f4ea96e9 started at 2022-11-23 23:29:44 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:37.290: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:32:37.290: INFO: pod-configmaps-6cefcfc7-988c-4c36-a7f4-05776840e9d1 started at 2022-11-23 23:31:22 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:37.290: INFO: Container createcm-volume-test ready: true, restart count 0 Nov 23 23:32:37.290: INFO: Container delcm-volume-test ready: true, restart count 0 Nov 23 23:32:37.290: INFO: Container updcm-volume-test ready: true, restart count 0 Nov 23 23:32:37.290: INFO: netserver-2 started at 2022-11-23 23:32:34 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:37.290: INFO: Container webserver ready: false, restart count 0 Nov 23 23:32:37.290: INFO: cilium-2lrb9 started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:32:37.290: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:32:37.290: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:32:37.290: INFO: startup-b399f159-4872-469e-94c2-952dd0185871 started at 2022-11-23 23:29:22 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:37.290: INFO: Container busybox ready: false, restart count 0 Nov 23 23:32:37.290: INFO: csi-mockplugin-0 started at 2022-11-23 23:31:52 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:37.290: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:32:37.290: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:32:37.290: INFO: Container mock ready: true, restart count 0 Nov 23 23:32:37.290: INFO: ebs-csi-node-dl575 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:37.290: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:37.290: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:37.290: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:37.860: INFO: Latency metrics for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:32:37.860: INFO: Logging node info for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:32:38.011: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-63-80.ap-northeast-1.compute.internal 1d4cdbf2-ca1f-431f-b562-a0bd0f489514 24070 0 2022-11-23 23:18:44 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-63-80.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-63-80.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-4069":"csi-mock-csi-mock-volumes-4069","ebs.csi.aws.com":"i-081735fbd2bce39f2"} io.cilium.network.ipv4-cilium-host:172.20.50.140 io.cilium.network.ipv4-health-ip:172.20.42.94 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:19:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:19:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:31:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-081735fbd2bce39f2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:19:12 +0000 UTC,LastTransitionTime:2022-11-23 23:19:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:54 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:54 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:54 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:31:54 +0000 UTC,LastTransitionTime:2022-11-23 23:19:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.63.80,},NodeAddress{Type:ExternalIP,Address:13.112.241.172,},NodeAddress{Type:Hostname,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-112-241-172.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29233394293c3c1bf3f4359c382b54,SystemUUID:ec292333-9429-3c3c-1bf3-f4359c382b54,BootID:05304de1-80b3-439c-9eeb-74d1d733f407,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:32:38.012: INFO: Logging kubelet events for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:32:38.166: INFO: Logging pods the kubelet thinks is on node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:32:38.321: INFO: cilium-dgm5r started at 2022-11-23 23:18:44 +0000 UTC (1+1 container statuses recorded) Nov 23 23:32:38.322: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:32:38.322: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:32:38.322: INFO: netserver-3 started at 2022-11-23 23:32:34 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:38.322: INFO: Container webserver ready: false, restart count 0 Nov 23 23:32:38.322: INFO: test-webserver-21ab2936-d38b-440a-98d8-5b0113a1ffbc started at 2022-11-23 23:31:23 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:38.322: INFO: Container test-webserver ready: true, restart count 0 Nov 23 23:32:38.322: INFO: csi-mockplugin-0 started at 2022-11-23 23:32:15 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:38.322: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:32:38.322: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:32:38.322: INFO: Container mock ready: true, restart count 0 Nov 23 23:32:38.322: INFO: ebs-csi-node-hlgh2 started at 2022-11-23 23:18:44 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:38.322: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:38.322: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:38.322: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:38.322: INFO: liveness-183c7acb-0cfd-4281-8b21-0f7d3cc95769 started at 2022-11-23 23:31:14 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:38.322: INFO: Container agnhost-container ready: true, restart count 4 Nov 23 23:32:38.322: INFO: netserver-3 started at 2022-11-23 23:32:16 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:38.322: INFO: Container webserver ready: true, restart count 0 Nov 23 23:32:38.322: INFO: csi-mockplugin-resizer-0 started at 2022-11-23 23:32:15 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:38.322: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:32:38.876: INFO: Latency metrics for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:32:38.876: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "volume-expand-5399" for this suite.
Find pod-bdad766a-25d1-423f-a111-7df3bb88031a mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(Always\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\snew\spod\sfsgroup\sapplied\sto\svolume\scontents$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Nov 23 23:30:14.406: Unexpected error: <*errors.errorString | 0xc003a589c0>: { s: "pod \"pod-7fbc79fe-4e78-46a5-8a98-29424315e6aa\" is not Running: timed out waiting for the condition", } pod "pod-7fbc79fe-4e78-46a5-8a98-29424315e6aa" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250from junit_25.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Nov 23 23:25:12.553: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename fsgroupchangepolicy �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Nov 23 23:25:13.323: INFO: Creating resource for dynamic PV Nov 23 23:25:13.323: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass fsgroupchangepolicy-9758-e2e-scs48xg �[1mSTEP�[0m: creating a claim Nov 23 23:25:13.478: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-9758 with fsgroup 1000 Nov 23 23:30:14.406: FAIL: Unexpected error: <*errors.errorString | 0xc003a589c0>: { s: "pod \"pod-7fbc79fe-4e78-46a5-8a98-29424315e6aa\" is not Running: timed out waiting for the condition", } pod "pod-7fbc79fe-4e78-46a5-8a98-29424315e6aa" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc0043bc000, 0xc0031650d8, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250 +0x1c5 k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:221 +0x219 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000903380) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000903380) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000903380, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 �[1mSTEP�[0m: Deleting pvc Nov 23 23:30:14.559: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.com78jw9" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "fsgroupchangepolicy-9758". �[1mSTEP�[0m: Found 5 events. Nov 23 23:30:15.022: INFO: At 2022-11-23 23:25:13 +0000 UTC - event for ebs.csi.aws.com78jw9: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 23:30:15.022: INFO: At 2022-11-23 23:25:13 +0000 UTC - event for ebs.csi.aws.com78jw9: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 23:30:15.022: INFO: At 2022-11-23 23:25:13 +0000 UTC - event for ebs.csi.aws.com78jw9: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-9758/ebs.csi.aws.com78jw9" Nov 23 23:30:15.022: INFO: At 2022-11-23 23:25:23 +0000 UTC - event for ebs.csi.aws.com78jw9: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-9758-e2e-scs48xg": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 23:30:15.022: INFO: At 2022-11-23 23:30:14 +0000 UTC - event for pod-7fbc79fe-4e78-46a5-8a98-29424315e6aa: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-9758/ebs.csi.aws.com78jw9" Nov 23 23:30:15.175: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 23:30:15.175: INFO: pod-7fbc79fe-4e78-46a5-8a98-29424315e6aa Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2022-11-23 23:30:14 +0000 UTC SchedulerError running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-9758/ebs.csi.aws.com78jw9"}] Nov 23 23:30:15.175: INFO: Nov 23 23:30:15.328: INFO: Logging node info for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:30:15.481: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-139.ap-northeast-1.compute.internal e52821e5-e82f-478b-be4f-6a60ba5816ce 18023 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-139.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-4383 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-35-139.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-4285":"csi-mock-csi-mock-volumes-4285","csi-mock-csi-mock-volumes-4950":"csi-mock-csi-mock-volumes-4950","ebs.csi.aws.com":"i-0af5c992eacc11f5f"} io.cilium.network.ipv4-cilium-host:172.20.62.73 io.cilium.network.ipv4-health-ip:172.20.63.125 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-11-23 23:27:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:27:56 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 23:28:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0af5c992eacc11f5f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:51 +0000 UTC,LastTransitionTime:2022-11-23 23:18:51 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:30:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:30:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:30:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:30:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.139,},NodeAddress{Type:ExternalIP,Address:13.231.193.129,},NodeAddress{Type:Hostname,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-231-193-129.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec218b5e615d578e61275cbc46e5a775,SystemUUID:ec218b5e-615d-578e-6127-5cbc46e5a775,BootID:15a3520e-c72a-46fe-a3d6-be039aa283b7,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-4950^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-4950^4,DevicePath:,},},Config:nil,},} Nov 23 23:30:15.481: INFO: Logging kubelet events for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:30:15.636: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:30:15.801: INFO: cilium-dw7qk started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:30:15.801: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:30:15.801: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:30:15.801: INFO: ebs-csi-node-vbdc7 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:15.801: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:30:15.801: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:30:15.801: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:30:15.801: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:29:50 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:15.801: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:30:15.801: INFO: csi-mockplugin-0 started at 2022-11-23 23:27:48 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:15.801: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:30:15.801: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:30:15.801: INFO: Container mock ready: true, restart count 0 Nov 23 23:30:15.801: INFO: csi-mockplugin-0 started at 2022-11-23 23:29:49 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:15.801: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:30:15.801: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:30:15.801: INFO: Container mock ready: true, restart count 0 Nov 23 23:30:15.801: INFO: csi-mockplugin-0 started at 2022-11-23 23:30:14 +0000 UTC (0+4 container statuses recorded) Nov 23 23:30:15.801: INFO: Container busybox ready: false, restart count 0 Nov 23 23:30:15.801: INFO: Container csi-provisioner ready: false, restart count 0 Nov 23 23:30:15.801: INFO: Container driver-registrar ready: false, restart count 0 Nov 23 23:30:15.801: INFO: Container mock ready: false, restart count 0 Nov 23 23:30:15.801: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:27:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:15.801: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:30:16.351: INFO: Latency metrics for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:30:16.351: INFO: Logging node info for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:30:16.504: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-241.ap-northeast-1.compute.internal 86ed4d71-4560-48af-9e88-d8d85c882f8b 16487 0 2022-11-23 23:16:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-241.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-027602f60f12db3f0"} io.cilium.network.ipv4-cilium-host:172.20.42.20 io.cilium.network.ipv4-health-ip:172.20.32.68 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 23:16:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-11-23 23:16:55 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-11-23 23:17:04 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:17:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 23:18:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:18:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-027602f60f12db3f0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892301824 0} {<nil>} 3801076Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787444224 0} {<nil>} 3698676Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:42 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:17:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.241,},NodeAddress{Type:ExternalIP,Address:54.238.84.104,},NodeAddress{Type:Hostname,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-84-104.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2adfc78fc274df02e3c085caa34395,SystemUUID:ec2adfc7-8fc2-74df-02e3-c085caa34395,BootID:e3997507-167a-4ec7-a6ec-7f97c5d886f5,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42904862,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41478108,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955301,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:30:16.504: INFO: Logging kubelet events for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:30:16.659: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:30:16.817: INFO: cilium-qvdj2 started at 2022-11-23 23:17:02 +0000 UTC (1+1 container statuses recorded) Nov 23 23:30:16.817: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:30:16.817: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 23:30:16.817: INFO: cilium-operator-587dd66f87-j2km7 started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:16.817: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 23:30:16.817: INFO: ebs-csi-controller-764b64d679-r8ttz started at 2022-11-23 23:17:03 +0000 UTC (0+5 container statuses recorded) Nov 23 23:30:16.817: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:30:16.817: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:30:16.817: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:30:16.817: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:30:16.817: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:30:16.817: INFO: etcd-manager-events-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:16.817: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:30:16.817: INFO: kube-scheduler-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:16.817: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 23:30:16.817: INFO: etcd-manager-main-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:16.817: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:30:16.817: INFO: kube-apiserver-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+2 container statuses recorded) Nov 23 23:30:16.817: INFO: Container healthcheck ready: true, restart count 0 Nov 23 23:30:16.817: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 23:30:16.817: INFO: ebs-csi-node-94jwj started at 2022-11-23 23:17:02 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:16.817: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:30:16.817: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:30:16.817: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:30:16.817: INFO: kops-controller-fznlr started at 2022-11-23 23:17:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:16.817: INFO: Container kops-controller ready: true, restart count 0 Nov 23 23:30:16.817: INFO: dns-controller-845449865f-zwg9n started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:16.817: INFO: Container dns-controller ready: true, restart count 0 Nov 23 23:30:16.817: INFO: kube-controller-manager-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:16.817: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 23:30:17.358: INFO: Latency metrics for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:30:17.358: INFO: Logging node info for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:30:17.516: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-42-64.ap-northeast-1.compute.internal 45916c04-689e-47fa-b6fb-bdbad0784fcd 18170 0 2022-11-23 23:18:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-42-64.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-42-64.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-8059":"csi-mock-csi-mock-volumes-8059","ebs.csi.aws.com":"i-049ec10a59a1d20e4"} io.cilium.network.ipv4-cilium-host:172.20.51.104 io.cilium.network.ipv4-health-ip:172.20.34.63 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:29:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-049ec10a59a1d20e4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:56 +0000 UTC,LastTransitionTime:2022-11-23 23:18:56 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:29:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.64,},NodeAddress{Type:ExternalIP,Address:54.238.96.164,},NodeAddress{Type:Hostname,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-96-164.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25c6eb725883a2c2916889bd12be88,SystemUUID:ec25c6eb-7258-83a2-c291-6889bd12be88,BootID:8214bf1e-ee6e-4136-8902-31d30749be64,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:30:17.517: INFO: Logging kubelet events for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:30:17.672: INFO: Logging pods the kubelet thinks is on node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:30:17.832: INFO: hostexec-ip-172-20-42-64.ap-northeast-1.compute.internal-cb74x started at 2022-11-23 23:30:10 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:17.832: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:30:17.832: INFO: hostexec-ip-172-20-42-64.ap-northeast-1.compute.internal-t2tt4 started at 2022-11-23 23:30:11 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:17.832: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:30:17.832: INFO: csi-mockplugin-resizer-0 started at 2022-11-23 23:29:54 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:17.832: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:30:17.832: INFO: cilium-jjmjs started at 2022-11-23 23:18:29 +0000 UTC (1+1 container statuses recorded) Nov 23 23:30:17.832: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:30:17.832: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:30:17.832: INFO: csi-mockplugin-0 started at 2022-11-23 23:29:54 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:17.832: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:30:17.832: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:30:17.832: INFO: Container mock ready: true, restart count 0 Nov 23 23:30:17.832: INFO: hostexec-ip-172-20-42-64.ap-northeast-1.compute.internal-h8h8g started at 2022-11-23 23:29:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:17.832: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:30:17.832: INFO: pod-cbabaf19-84c6-44b6-946e-45a57cec7d25 started at <nil> (0+0 container statuses recorded) Nov 23 23:30:17.832: INFO: local-injector started at 2022-11-23 23:30:04 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:17.832: INFO: Container local-injector ready: true, restart count 0 Nov 23 23:30:17.832: INFO: coredns-5fcc7b6498-s4jjd started at 2022-11-23 23:19:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:17.832: INFO: Container coredns ready: true, restart count 0 Nov 23 23:30:17.832: INFO: hostexec-ip-172-20-42-64.ap-northeast-1.compute.internal-2zdds started at 2022-11-23 23:29:55 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:17.832: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:30:17.832: INFO: ebs-csi-node-fq5pw started at 2022-11-23 23:18:29 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:17.832: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:30:17.832: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:30:17.832: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:30:18.404: INFO: Latency metrics for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:30:18.404: INFO: Logging node info for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:30:18.557: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-200.ap-northeast-1.compute.internal 8f32a357-4fd8-4fcd-ab21-8b69dbd931d6 13785 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-200.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-6944":"csi-mock-csi-mock-volumes-6944","ebs.csi.aws.com":"i-0d4c764fea6a24b23"} io.cilium.network.ipv4-cilium-host:172.20.61.17 io.cilium.network.ipv4-health-ip:172.20.60.108 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:50 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:24:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0d4c764fea6a24b23,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:50 +0000 UTC,LastTransitionTime:2022-11-23 23:18:50 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.200,},NodeAddress{Type:ExternalIP,Address:3.113.30.221,},NodeAddress{Type:Hostname,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-113-30-221.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ef1c5e899b01a14fef22e61702fe5,SystemUUID:ec2ef1c5-e899-b01a-14fe-f22e61702fe5,BootID:021f6d58-0800-4a46-b65e-3513c48dff08,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:30:18.557: INFO: Logging kubelet events for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:30:18.712: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:30:18.869: INFO: ebs-csi-node-dl575 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:18.869: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:30:18.869: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:30:18.869: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:30:18.869: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:27:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:18.869: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:30:18.869: INFO: cilium-2lrb9 started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:30:18.869: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:30:18.869: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:30:18.869: INFO: coredns-autoscaler-6658b4bf85-gdgxf started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:18.869: INFO: Container autoscaler ready: true, restart count 0 Nov 23 23:30:18.869: INFO: coredns-5fcc7b6498-ll578 started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:18.869: INFO: Container coredns ready: true, restart count 0 Nov 23 23:30:18.869: INFO: liveness-c01c2deb-84cd-4e87-b47a-aab4f4ea96e9 started at 2022-11-23 23:29:44 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:18.869: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:30:18.869: INFO: csi-mockplugin-0 started at 2022-11-23 23:27:49 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:18.869: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:30:18.869: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:30:18.869: INFO: Container mock ready: true, restart count 0 Nov 23 23:30:18.869: INFO: startup-b399f159-4872-469e-94c2-952dd0185871 started at 2022-11-23 23:29:22 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:18.869: INFO: Container busybox ready: false, restart count 0 Nov 23 23:30:19.377: INFO: Latency metrics for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:30:19.377: INFO: Logging node info for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:30:19.530: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-63-80.ap-northeast-1.compute.internal 1d4cdbf2-ca1f-431f-b562-a0bd0f489514 18199 0 2022-11-23 23:18:44 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-63-80.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-63-80.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-081735fbd2bce39f2"} io.cilium.network.ipv4-cilium-host:172.20.50.140 io.cilium.network.ipv4-health-ip:172.20.42.94 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:19:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:19:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:30:04 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-081735fbd2bce39f2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:19:12 +0000 UTC,LastTransitionTime:2022-11-23 23:19:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:30:14 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:30:14 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:30:14 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:30:14 +0000 UTC,LastTransitionTime:2022-11-23 23:19:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.63.80,},NodeAddress{Type:ExternalIP,Address:13.112.241.172,},NodeAddress{Type:Hostname,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-112-241-172.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29233394293c3c1bf3f4359c382b54,SystemUUID:ec292333-9429-3c3c-1bf3-f4359c382b54,BootID:05304de1-80b3-439c-9eeb-74d1d733f407,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:30:19.530: INFO: Logging kubelet events for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:30:19.687: INFO: Logging pods the kubelet thinks is on node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:30:19.907: INFO: cilium-dgm5r started at 2022-11-23 23:18:44 +0000 UTC (1+1 container statuses recorded) Nov 23 23:30:19.907: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:30:19.907: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:30:19.907: INFO: ebs-csi-node-hlgh2 started at 2022-11-23 23:18:44 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:19.907: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:30:19.907: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:30:19.907: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:30:19.907: INFO: pod-configmaps-96b54f3c-70bd-4fc0-8728-dfc86cdffbce started at 2022-11-23 23:30:15 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:19.907: INFO: Container agnhost-container ready: false, restart count 0 Nov 23 23:30:20.416: INFO: Latency metrics for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:30:20.416: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "fsgroupchangepolicy-9758" for this suite.
Find pod-7fbc79fe-4e78-46a5-8a98-29424315e6aa mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(Always\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\svolume\scontents\sownership\schanged\sin\sfirst\spod\,\snew\spod\swith\sdifferent\sfsgroup\sapplied\sto\sthe\svolume\scontents$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Nov 23 23:32:12.279: Unexpected error: <*errors.errorString | 0xc000cbe730>: { s: "pod \"pod-031b0f7d-d197-4569-a359-9dc115762812\" is not Running: timed out waiting for the condition", } pod "pod-031b0f7d-d197-4569-a359-9dc115762812" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250from junit_06.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Nov 23 23:27:10.396: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename fsgroupchangepolicy �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with different fsgroup applied to the volume contents /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Nov 23 23:27:11.187: INFO: Creating resource for dynamic PV Nov 23 23:27:11.187: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass fsgroupchangepolicy-3225-e2e-sckk2j6 �[1mSTEP�[0m: creating a claim Nov 23 23:27:11.344: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-3225 with fsgroup 1000 Nov 23 23:32:12.279: FAIL: Unexpected error: <*errors.errorString | 0xc000cbe730>: { s: "pod \"pod-031b0f7d-d197-4569-a359-9dc115762812\" is not Running: timed out waiting for the condition", } pod "pod-031b0f7d-d197-4569-a359-9dc115762812" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc003d6f1e0, 0xc0032af0d8, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250 +0x1c5 k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:221 +0x219 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000ccc180) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000ccc180) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000ccc180, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 �[1mSTEP�[0m: Deleting pvc Nov 23 23:32:12.435: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.com6lhq7" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "fsgroupchangepolicy-3225". �[1mSTEP�[0m: Found 6 events. Nov 23 23:32:12.901: INFO: At 2022-11-23 23:27:11 +0000 UTC - event for ebs.csi.aws.com6lhq7: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 23:32:12.901: INFO: At 2022-11-23 23:27:11 +0000 UTC - event for ebs.csi.aws.com6lhq7: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 23:32:12.901: INFO: At 2022-11-23 23:27:11 +0000 UTC - event for ebs.csi.aws.com6lhq7: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-3225/ebs.csi.aws.com6lhq7" Nov 23 23:32:12.901: INFO: At 2022-11-23 23:27:21 +0000 UTC - event for ebs.csi.aws.com6lhq7: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-3225-e2e-sckk2j6": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 23:32:12.901: INFO: At 2022-11-23 23:28:42 +0000 UTC - event for ebs.csi.aws.com6lhq7: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-3225-e2e-sckk2j6": rpc error: code = Internal desc = Could not create volume "pvc-224f62d1-6c68-4a36-a40f-4244cdc1cd5d": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Nov 23 23:32:12.901: INFO: At 2022-11-23 23:32:12 +0000 UTC - event for pod-031b0f7d-d197-4569-a359-9dc115762812: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-3225/ebs.csi.aws.com6lhq7" Nov 23 23:32:13.056: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 23:32:13.056: INFO: pod-031b0f7d-d197-4569-a359-9dc115762812 Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2022-11-23 23:32:12 +0000 UTC SchedulerError running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-3225/ebs.csi.aws.com6lhq7"}] Nov 23 23:32:13.056: INFO: Nov 23 23:32:13.212: INFO: Logging node info for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:32:13.367: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-139.ap-northeast-1.compute.internal e52821e5-e82f-478b-be4f-6a60ba5816ce 21725 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-139.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-4383 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-35-139.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0af5c992eacc11f5f"} io.cilium.network.ipv4-cilium-host:172.20.62.73 io.cilium.network.ipv4-health-ip:172.20.63.125 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-11-23 23:27:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kubelet Update v1 2022-11-23 23:31:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0af5c992eacc11f5f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:51 +0000 UTC,LastTransitionTime:2022-11-23 23:18:51 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.139,},NodeAddress{Type:ExternalIP,Address:13.231.193.129,},NodeAddress{Type:Hostname,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-231-193-129.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec218b5e615d578e61275cbc46e5a775,SystemUUID:ec218b5e-615d-578e-6127-5cbc46e5a775,BootID:15a3520e-c72a-46fe-a3d6-be039aa283b7,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:32:13.367: INFO: Logging kubelet events for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:32:13.525: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:32:13.872: INFO: netserver-0 started at 2022-11-23 23:31:44 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:13.872: INFO: Container webserver ready: true, restart count 0 Nov 23 23:32:13.872: INFO: webserver-deployment-847dcfb7fb-xt65n started at 2022-11-23 23:32:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:13.872: INFO: Container httpd ready: true, restart count 0 Nov 23 23:32:13.872: INFO: webserver-deployment-847dcfb7fb-bw5h2 started at 2022-11-23 23:32:09 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:13.872: INFO: Container httpd ready: true, restart count 0 Nov 23 23:32:13.872: INFO: webserver-deployment-795d758f88-x6cj7 started at 2022-11-23 23:32:09 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:13.872: INFO: Container httpd ready: false, restart count 0 Nov 23 23:32:13.872: INFO: webserver-deployment-795d758f88-x62d9 started at 2022-11-23 23:32:09 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:13.872: INFO: Container httpd ready: false, restart count 0 Nov 23 23:32:13.872: INFO: dns-test-061c5190-7657-46b0-9ec1-d8e522d50321 started at 2022-11-23 23:31:58 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:13.872: INFO: Container jessie-querier ready: true, restart count 0 Nov 23 23:32:13.872: INFO: Container querier ready: true, restart count 0 Nov 23 23:32:13.872: INFO: Container webserver ready: true, restart count 0 Nov 23 23:32:13.872: INFO: webserver-deployment-847dcfb7fb-82xdq started at 2022-11-23 23:32:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:13.872: INFO: Container httpd ready: true, restart count 0 Nov 23 23:32:13.872: INFO: host-test-container-pod started at 2022-11-23 23:32:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:13.872: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:32:13.872: INFO: webserver-deployment-795d758f88-vp8th started at 2022-11-23 23:32:07 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:13.872: INFO: Container httpd ready: false, restart count 0 Nov 23 23:32:13.872: INFO: cilium-dw7qk started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:32:13.872: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:32:13.872: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:32:13.872: INFO: webserver-deployment-847dcfb7fb-sjvtx started at 2022-11-23 23:32:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:13.872: INFO: Container httpd ready: true, restart count 0 Nov 23 23:32:13.872: INFO: test-container-pod started at 2022-11-23 23:32:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:13.872: INFO: Container webserver ready: true, restart count 0 Nov 23 23:32:13.872: INFO: webserver-deployment-847dcfb7fb-jmnvp started at 2022-11-23 23:32:09 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:13.872: INFO: Container httpd ready: false, restart count 0 Nov 23 23:32:13.872: INFO: ebs-csi-node-vbdc7 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:13.872: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:13.872: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:13.872: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:13.872: INFO: netserver-0 started at 2022-11-23 23:31:41 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:13.872: INFO: Container webserver ready: true, restart count 0 Nov 23 23:32:13.872: INFO: webserver-deployment-795d758f88-xlft7 started at 2022-11-23 23:32:09 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:13.872: INFO: Container httpd ready: false, restart count 0 Nov 23 23:32:14.426: INFO: Latency metrics for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:32:14.426: INFO: Logging node info for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:32:14.582: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-241.ap-northeast-1.compute.internal 86ed4d71-4560-48af-9e88-d8d85c882f8b 16487 0 2022-11-23 23:16:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-241.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-027602f60f12db3f0"} io.cilium.network.ipv4-cilium-host:172.20.42.20 io.cilium.network.ipv4-health-ip:172.20.32.68 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 23:16:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-11-23 23:16:55 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-11-23 23:17:04 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:17:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 23:18:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:18:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-027602f60f12db3f0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892301824 0} {<nil>} 3801076Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787444224 0} {<nil>} 3698676Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:42 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:17:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.241,},NodeAddress{Type:ExternalIP,Address:54.238.84.104,},NodeAddress{Type:Hostname,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-84-104.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2adfc78fc274df02e3c085caa34395,SystemUUID:ec2adfc7-8fc2-74df-02e3-c085caa34395,BootID:e3997507-167a-4ec7-a6ec-7f97c5d886f5,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42904862,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41478108,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955301,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:32:14.582: INFO: Logging kubelet events for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:32:14.741: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:32:14.920: INFO: kube-scheduler-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:14.920: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 23:32:14.920: INFO: etcd-manager-main-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:14.920: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:32:14.920: INFO: kube-apiserver-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+2 container statuses recorded) Nov 23 23:32:14.920: INFO: Container healthcheck ready: true, restart count 0 Nov 23 23:32:14.920: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 23:32:14.920: INFO: ebs-csi-node-94jwj started at 2022-11-23 23:17:02 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:14.920: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:14.920: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:14.920: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:14.920: INFO: kops-controller-fznlr started at 2022-11-23 23:17:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:14.920: INFO: Container kops-controller ready: true, restart count 0 Nov 23 23:32:14.920: INFO: dns-controller-845449865f-zwg9n started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:14.920: INFO: Container dns-controller ready: true, restart count 0 Nov 23 23:32:14.920: INFO: kube-controller-manager-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:14.920: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 23:32:14.920: INFO: cilium-qvdj2 started at 2022-11-23 23:17:02 +0000 UTC (1+1 container statuses recorded) Nov 23 23:32:14.920: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:32:14.920: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 23:32:14.920: INFO: cilium-operator-587dd66f87-j2km7 started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:14.920: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 23:32:14.920: INFO: ebs-csi-controller-764b64d679-r8ttz started at 2022-11-23 23:17:03 +0000 UTC (0+5 container statuses recorded) Nov 23 23:32:14.920: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:32:14.920: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:32:14.920: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:32:14.920: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:14.920: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:14.920: INFO: etcd-manager-events-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:14.920: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:32:15.463: INFO: Latency metrics for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:32:15.463: INFO: Logging node info for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:32:15.619: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-42-64.ap-northeast-1.compute.internal 45916c04-689e-47fa-b6fb-bdbad0784fcd 23482 0 2022-11-23 23:18:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-42-64.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-42-64.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-6012":"csi-mock-csi-mock-volumes-6012","ebs.csi.aws.com":"i-049ec10a59a1d20e4"} io.cilium.network.ipv4-cilium-host:172.20.51.104 io.cilium.network.ipv4-health-ip:172.20.34.63 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:32:09 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-049ec10a59a1d20e4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:56 +0000 UTC,LastTransitionTime:2022-11-23 23:18:56 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:32:09 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:32:09 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:32:09 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:32:09 +0000 UTC,LastTransitionTime:2022-11-23 23:18:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.64,},NodeAddress{Type:ExternalIP,Address:54.238.96.164,},NodeAddress{Type:Hostname,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-96-164.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25c6eb725883a2c2916889bd12be88,SystemUUID:ec25c6eb-7258-83a2-c291-6889bd12be88,BootID:8214bf1e-ee6e-4136-8902-31d30749be64,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:32:15.619: INFO: Logging kubelet events for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:32:15.816: INFO: Logging pods the kubelet thinks is on node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:32:16.254: INFO: webserver-deployment-847dcfb7fb-pzrg6 started at 2022-11-23 23:32:09 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:16.254: INFO: Container httpd ready: true, restart count 0 Nov 23 23:32:16.254: INFO: coredns-5fcc7b6498-s4jjd started at 2022-11-23 23:19:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:16.254: INFO: Container coredns ready: true, restart count 0 Nov 23 23:32:16.254: INFO: webserver-deployment-847dcfb7fb-c5b5x started at 2022-11-23 23:32:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:16.254: INFO: Container httpd ready: true, restart count 0 Nov 23 23:32:16.254: INFO: hostexec-ip-172-20-42-64.ap-northeast-1.compute.internal-qt7ld started at 2022-11-23 23:31:56 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:16.254: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:32:16.254: INFO: test-container-pod started at 2022-11-23 23:32:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:16.254: INFO: Container webserver ready: true, restart count 0 Nov 23 23:32:16.254: INFO: webserver-deployment-795d758f88-ksccn started at 2022-11-23 23:32:09 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:16.254: INFO: Container httpd ready: false, restart count 0 Nov 23 23:32:16.254: INFO: ebs-csi-node-fq5pw started at 2022-11-23 23:18:29 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:16.254: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:16.254: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:16.254: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:16.254: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:31:46 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:16.254: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:32:16.254: INFO: webserver-deployment-847dcfb7fb-7x8q4 started at 2022-11-23 23:32:09 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:16.254: INFO: Container httpd ready: true, restart count 0 Nov 23 23:32:16.254: INFO: webserver-deployment-847dcfb7fb-fmxjh started at 2022-11-23 23:32:09 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:16.254: INFO: Container httpd ready: true, restart count 0 Nov 23 23:32:16.254: INFO: cilium-jjmjs started at 2022-11-23 23:18:29 +0000 UTC (1+1 container statuses recorded) Nov 23 23:32:16.254: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:32:16.254: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:32:16.254: INFO: csi-mockplugin-0 started at 2022-11-23 23:31:46 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:16.254: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:32:16.254: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:32:16.254: INFO: Container mock ready: true, restart count 0 Nov 23 23:32:16.254: INFO: webserver-deployment-847dcfb7fb-7rtf8 started at 2022-11-23 23:32:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:16.254: INFO: Container httpd ready: true, restart count 0 Nov 23 23:32:17.516: INFO: Latency metrics for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:32:17.516: INFO: Logging node info for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:32:17.671: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-200.ap-northeast-1.compute.internal 8f32a357-4fd8-4fcd-ab21-8b69dbd931d6 23091 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-200.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9079":"csi-mock-csi-mock-volumes-9079","ebs.csi.aws.com":"i-0d4c764fea6a24b23"} io.cilium.network.ipv4-cilium-host:172.20.61.17 io.cilium.network.ipv4-health-ip:172.20.60.108 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:50 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:24:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0d4c764fea6a24b23,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:50 +0000 UTC,LastTransitionTime:2022-11-23 23:18:50 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.200,},NodeAddress{Type:ExternalIP,Address:3.113.30.221,},NodeAddress{Type:Hostname,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-113-30-221.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ef1c5e899b01a14fef22e61702fe5,SystemUUID:ec2ef1c5-e899-b01a-14fe-f22e61702fe5,BootID:021f6d58-0800-4a46-b65e-3513c48dff08,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:32:17.672: INFO: Logging kubelet events for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:32:17.829: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:32:18.589: INFO: csi-mockplugin-0 started at 2022-11-23 23:31:52 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:18.589: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:32:18.589: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:32:18.589: INFO: Container mock ready: true, restart count 0 Nov 23 23:32:18.589: INFO: hostpath-client started at 2022-11-23 23:32:12 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:18.589: INFO: Container hostpath-client ready: true, restart count 0 Nov 23 23:32:18.589: INFO: cilium-2lrb9 started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:32:18.589: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:32:18.589: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:32:18.589: INFO: startup-b399f159-4872-469e-94c2-952dd0185871 started at 2022-11-23 23:29:22 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:18.589: INFO: Container busybox ready: false, restart count 0 Nov 23 23:32:18.589: INFO: ebs-csi-node-dl575 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:18.589: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:18.589: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:18.589: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:18.589: INFO: pod-subpath-test-configmap-6ln4 started at 2022-11-23 23:30:55 +0000 UTC (1+2 container statuses recorded) Nov 23 23:32:18.589: INFO: Init container init-volume-configmap-6ln4 ready: true, restart count 0 Nov 23 23:32:18.589: INFO: Container test-container-subpath-configmap-6ln4 ready: true, restart count 3 Nov 23 23:32:18.589: INFO: Container test-container-volume-configmap-6ln4 ready: true, restart count 0 Nov 23 23:32:18.589: INFO: pod-configmaps-6cefcfc7-988c-4c36-a7f4-05776840e9d1 started at 2022-11-23 23:31:22 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:18.589: INFO: Container createcm-volume-test ready: true, restart count 0 Nov 23 23:32:18.589: INFO: Container delcm-volume-test ready: true, restart count 0 Nov 23 23:32:18.589: INFO: Container updcm-volume-test ready: true, restart count 0 Nov 23 23:32:18.589: INFO: netserver-2 started at <nil> (0+0 container statuses recorded) Nov 23 23:32:18.589: INFO: pvc-volume-tester-qtr6f started at 2022-11-23 23:32:04 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:18.589: INFO: Container volume-tester ready: false, restart count 0 Nov 23 23:32:18.589: INFO: coredns-autoscaler-6658b4bf85-gdgxf started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:18.589: INFO: Container autoscaler ready: true, restart count 0 Nov 23 23:32:18.589: INFO: coredns-5fcc7b6498-ll578 started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:18.589: INFO: Container coredns ready: true, restart count 0 Nov 23 23:32:18.589: INFO: liveness-c01c2deb-84cd-4e87-b47a-aab4f4ea96e9 started at 2022-11-23 23:29:44 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:18.589: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:32:19.644: INFO: Latency metrics for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:32:19.644: INFO: Logging node info for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:32:19.800: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-63-80.ap-northeast-1.compute.internal 1d4cdbf2-ca1f-431f-b562-a0bd0f489514 23468 0 2022-11-23 23:18:44 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-63-80.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-63-80.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-081735fbd2bce39f2"} io.cilium.network.ipv4-cilium-host:172.20.50.140 io.cilium.network.ipv4-health-ip:172.20.42.94 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:19:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:19:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:31:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-081735fbd2bce39f2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:19:12 +0000 UTC,LastTransitionTime:2022-11-23 23:19:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:54 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:54 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:54 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:31:54 +0000 UTC,LastTransitionTime:2022-11-23 23:19:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.63.80,},NodeAddress{Type:ExternalIP,Address:13.112.241.172,},NodeAddress{Type:Hostname,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-112-241-172.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29233394293c3c1bf3f4359c382b54,SystemUUID:ec292333-9429-3c3c-1bf3-f4359c382b54,BootID:05304de1-80b3-439c-9eeb-74d1d733f407,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:32:19.806: INFO: Logging kubelet events for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:32:19.965: INFO: Logging pods the kubelet thinks is on node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:32:20.449: INFO: csi-mockplugin-0 started at 2022-11-23 23:32:15 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:20.449: INFO: Container csi-provisioner ready: false, restart count 0 Nov 23 23:32:20.449: INFO: Container driver-registrar ready: false, restart count 0 Nov 23 23:32:20.449: INFO: Container mock ready: false, restart count 0 Nov 23 23:32:20.449: INFO: cilium-dgm5r started at 2022-11-23 23:18:44 +0000 UTC (1+1 container statuses recorded) Nov 23 23:32:20.449: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:32:20.449: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:32:20.449: INFO: test-webserver-21ab2936-d38b-440a-98d8-5b0113a1ffbc started at 2022-11-23 23:31:23 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:20.449: INFO: Container test-webserver ready: true, restart count 0 Nov 23 23:32:20.449: INFO: ebs-csi-node-hlgh2 started at 2022-11-23 23:18:44 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:20.449: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:20.449: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:20.449: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:20.449: INFO: liveness-183c7acb-0cfd-4281-8b21-0f7d3cc95769 started at 2022-11-23 23:31:14 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:20.449: INFO: Container agnhost-container ready: true, restart count 3 Nov 23 23:32:20.449: INFO: downwardapi-volume-b70a999b-b079-44c1-abcf-c3f8493482a0 started at 2022-11-23 23:32:11 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:20.449: INFO: Container client-container ready: false, restart count 0 Nov 23 23:32:20.449: INFO: netserver-3 started at <nil> (0+0 container statuses recorded) Nov 23 23:32:20.449: INFO: csi-mockplugin-resizer-0 started at 2022-11-23 23:32:15 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:20.449: INFO: Container csi-resizer ready: false, restart count 0 Nov 23 23:32:21.007: INFO: Latency metrics for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:32:21.007: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "fsgroupchangepolicy-3225" for this suite.
Find pod-031b0f7d-d197-4569-a359-9dc115762812 mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(Always\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\svolume\scontents\sownership\schanged\sin\sfirst\spod\,\snew\spod\swith\ssame\sfsgroup\sapplied\sto\sthe\svolume\scontents$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Nov 23 23:31:37.943: Unexpected error: <*errors.errorString | 0xc0037c7250>: { s: "pod \"pod-d1c4acff-661d-4cbd-9ca6-1bf2d835eb6a\" is not Running: timed out waiting for the condition", } pod "pod-d1c4acff-661d-4cbd-9ca6-1bf2d835eb6a" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250from junit_15.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Nov 23 23:26:36.100: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename fsgroupchangepolicy �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup applied to the volume contents /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Nov 23 23:26:36.867: INFO: Creating resource for dynamic PV Nov 23 23:26:36.867: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass fsgroupchangepolicy-8255-e2e-scxchwd �[1mSTEP�[0m: creating a claim Nov 23 23:26:37.022: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-8255 with fsgroup 1000 Nov 23 23:31:37.943: FAIL: Unexpected error: <*errors.errorString | 0xc0037c7250>: { s: "pod \"pod-d1c4acff-661d-4cbd-9ca6-1bf2d835eb6a\" is not Running: timed out waiting for the condition", } pod "pod-d1c4acff-661d-4cbd-9ca6-1bf2d835eb6a" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc0034d3600, 0xc001dab0d8, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250 +0x1c5 k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:221 +0x219 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc00096a900) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc00096a900) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc00096a900, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 �[1mSTEP�[0m: Deleting pvc Nov 23 23:31:38.096: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.commdl76" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "fsgroupchangepolicy-8255". �[1mSTEP�[0m: Found 6 events. Nov 23 23:31:38.593: INFO: At 2022-11-23 23:26:37 +0000 UTC - event for ebs.csi.aws.commdl76: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 23:31:38.593: INFO: At 2022-11-23 23:26:37 +0000 UTC - event for ebs.csi.aws.commdl76: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 23:31:38.593: INFO: At 2022-11-23 23:26:37 +0000 UTC - event for ebs.csi.aws.commdl76: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-8255/ebs.csi.aws.commdl76" Nov 23 23:31:38.593: INFO: At 2022-11-23 23:26:47 +0000 UTC - event for ebs.csi.aws.commdl76: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-8255-e2e-scxchwd": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 23:31:38.593: INFO: At 2022-11-23 23:28:50 +0000 UTC - event for ebs.csi.aws.commdl76: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-8255-e2e-scxchwd": rpc error: code = Internal desc = Could not create volume "pvc-7bf87afc-6cec-472c-8374-e7659189b983": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Nov 23 23:31:38.593: INFO: At 2022-11-23 23:31:38 +0000 UTC - event for pod-d1c4acff-661d-4cbd-9ca6-1bf2d835eb6a: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-8255/ebs.csi.aws.commdl76" Nov 23 23:31:38.749: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 23:31:38.749: INFO: pod-d1c4acff-661d-4cbd-9ca6-1bf2d835eb6a Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2022-11-23 23:31:38 +0000 UTC SchedulerError running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-8255/ebs.csi.aws.commdl76"}] Nov 23 23:31:38.749: INFO: Nov 23 23:31:38.903: INFO: Logging node info for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:31:39.057: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-139.ap-northeast-1.compute.internal e52821e5-e82f-478b-be4f-6a60ba5816ce 21725 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-139.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-4383 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-35-139.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0af5c992eacc11f5f"} io.cilium.network.ipv4-cilium-host:172.20.62.73 io.cilium.network.ipv4-health-ip:172.20.63.125 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-11-23 23:27:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kubelet Update v1 2022-11-23 23:31:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0af5c992eacc11f5f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:51 +0000 UTC,LastTransitionTime:2022-11-23 23:18:51 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.139,},NodeAddress{Type:ExternalIP,Address:13.231.193.129,},NodeAddress{Type:Hostname,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-231-193-129.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec218b5e615d578e61275cbc46e5a775,SystemUUID:ec218b5e-615d-578e-6127-5cbc46e5a775,BootID:15a3520e-c72a-46fe-a3d6-be039aa283b7,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:31:39.057: INFO: Logging kubelet events for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:31:39.212: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:31:39.374: INFO: cilium-dw7qk started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:31:39.374: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:31:39.374: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:31:39.374: INFO: condition-test-lwv9v started at 2022-11-23 23:31:38 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:39.374: INFO: Container httpd ready: false, restart count 0 Nov 23 23:31:39.374: INFO: ebs-csi-node-vbdc7 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:31:39.374: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:31:39.374: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:31:39.374: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:31:39.918: INFO: Latency metrics for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:31:39.918: INFO: Logging node info for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:31:40.071: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-241.ap-northeast-1.compute.internal 86ed4d71-4560-48af-9e88-d8d85c882f8b 16487 0 2022-11-23 23:16:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-241.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-027602f60f12db3f0"} io.cilium.network.ipv4-cilium-host:172.20.42.20 io.cilium.network.ipv4-health-ip:172.20.32.68 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 23:16:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-11-23 23:16:55 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-11-23 23:17:04 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:17:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 23:18:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:18:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-027602f60f12db3f0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892301824 0} {<nil>} 3801076Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787444224 0} {<nil>} 3698676Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:42 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:17:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.241,},NodeAddress{Type:ExternalIP,Address:54.238.84.104,},NodeAddress{Type:Hostname,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-84-104.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2adfc78fc274df02e3c085caa34395,SystemUUID:ec2adfc7-8fc2-74df-02e3-c085caa34395,BootID:e3997507-167a-4ec7-a6ec-7f97c5d886f5,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42904862,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41478108,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955301,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:31:40.071: INFO: Logging kubelet events for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:31:40.228: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:31:40.386: INFO: kube-controller-manager-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:40.386: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 23:31:40.386: INFO: kube-scheduler-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:40.386: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 23:31:40.386: INFO: etcd-manager-main-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:40.386: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:31:40.386: INFO: kube-apiserver-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+2 container statuses recorded) Nov 23 23:31:40.386: INFO: Container healthcheck ready: true, restart count 0 Nov 23 23:31:40.386: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 23:31:40.386: INFO: ebs-csi-node-94jwj started at 2022-11-23 23:17:02 +0000 UTC (0+3 container statuses recorded) Nov 23 23:31:40.386: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:31:40.386: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:31:40.386: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:31:40.386: INFO: kops-controller-fznlr started at 2022-11-23 23:17:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:40.386: INFO: Container kops-controller ready: true, restart count 0 Nov 23 23:31:40.386: INFO: dns-controller-845449865f-zwg9n started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:40.386: INFO: Container dns-controller ready: true, restart count 0 Nov 23 23:31:40.386: INFO: etcd-manager-events-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:40.386: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:31:40.386: INFO: cilium-qvdj2 started at 2022-11-23 23:17:02 +0000 UTC (1+1 container statuses recorded) Nov 23 23:31:40.386: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:31:40.386: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 23:31:40.386: INFO: cilium-operator-587dd66f87-j2km7 started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:40.386: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 23:31:40.386: INFO: ebs-csi-controller-764b64d679-r8ttz started at 2022-11-23 23:17:03 +0000 UTC (0+5 container statuses recorded) Nov 23 23:31:40.386: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:31:40.386: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:31:40.386: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:31:40.386: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:31:40.386: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:31:40.911: INFO: Latency metrics for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:31:40.911: INFO: Logging node info for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:31:41.064: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-42-64.ap-northeast-1.compute.internal 45916c04-689e-47fa-b6fb-bdbad0784fcd 18644 0 2022-11-23 23:18:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-42-64.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-42-64.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-049ec10a59a1d20e4"} io.cilium.network.ipv4-cilium-host:172.20.51.104 io.cilium.network.ipv4-health-ip:172.20.34.63 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:29:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-049ec10a59a1d20e4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:56 +0000 UTC,LastTransitionTime:2022-11-23 23:18:56 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:30:29 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:30:29 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:30:29 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:30:29 +0000 UTC,LastTransitionTime:2022-11-23 23:18:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.64,},NodeAddress{Type:ExternalIP,Address:54.238.96.164,},NodeAddress{Type:Hostname,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-96-164.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25c6eb725883a2c2916889bd12be88,SystemUUID:ec25c6eb-7258-83a2-c291-6889bd12be88,BootID:8214bf1e-ee6e-4136-8902-31d30749be64,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:31:41.065: INFO: Logging kubelet events for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:31:41.219: INFO: Logging pods the kubelet thinks is on node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:31:41.378: INFO: cilium-jjmjs started at 2022-11-23 23:18:29 +0000 UTC (1+1 container statuses recorded) Nov 23 23:31:41.378: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:31:41.378: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:31:41.378: INFO: hostexec-ip-172-20-42-64.ap-northeast-1.compute.internal-8j26b started at 2022-11-23 23:31:25 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:41.378: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:31:41.378: INFO: coredns-5fcc7b6498-s4jjd started at 2022-11-23 23:19:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:41.378: INFO: Container coredns ready: true, restart count 0 Nov 23 23:31:41.378: INFO: dns-test-b5e1277a-ccf7-4f07-92b1-c8f6ed9ed26c started at 2022-11-23 23:31:20 +0000 UTC (0+3 container statuses recorded) Nov 23 23:31:41.378: INFO: Container jessie-querier ready: true, restart count 0 Nov 23 23:31:41.378: INFO: Container querier ready: true, restart count 0 Nov 23 23:31:41.378: INFO: Container webserver ready: true, restart count 0 Nov 23 23:31:41.378: INFO: ebs-csi-node-fq5pw started at 2022-11-23 23:18:29 +0000 UTC (0+3 container statuses recorded) Nov 23 23:31:41.378: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:31:41.378: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:31:41.378: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:31:41.378: INFO: condition-test-mb79f started at 2022-11-23 23:31:38 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:41.378: INFO: Container httpd ready: true, restart count 0 Nov 23 23:31:44.580: INFO: Latency metrics for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:31:44.580: INFO: Logging node info for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:31:44.734: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-200.ap-northeast-1.compute.internal 8f32a357-4fd8-4fcd-ab21-8b69dbd931d6 18288 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-200.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d4c764fea6a24b23"} io.cilium.network.ipv4-cilium-host:172.20.61.17 io.cilium.network.ipv4-health-ip:172.20.60.108 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:50 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:24:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0d4c764fea6a24b23,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:50 +0000 UTC,LastTransitionTime:2022-11-23 23:18:50 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.200,},NodeAddress{Type:ExternalIP,Address:3.113.30.221,},NodeAddress{Type:Hostname,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-113-30-221.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ef1c5e899b01a14fef22e61702fe5,SystemUUID:ec2ef1c5-e899-b01a-14fe-f22e61702fe5,BootID:021f6d58-0800-4a46-b65e-3513c48dff08,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:31:44.735: INFO: Logging kubelet events for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:31:44.898: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:31:45.060: INFO: ebs-csi-node-dl575 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:31:45.060: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:31:45.060: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:31:45.060: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:31:45.060: INFO: pod-subpath-test-configmap-6ln4 started at 2022-11-23 23:30:55 +0000 UTC (1+2 container statuses recorded) Nov 23 23:31:45.060: INFO: Init container init-volume-configmap-6ln4 ready: true, restart count 0 Nov 23 23:31:45.060: INFO: Container test-container-subpath-configmap-6ln4 ready: true, restart count 3 Nov 23 23:31:45.060: INFO: Container test-container-volume-configmap-6ln4 ready: true, restart count 0 Nov 23 23:31:45.060: INFO: netserver-2 started at 2022-11-23 23:31:41 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:45.060: INFO: Container webserver ready: false, restart count 0 Nov 23 23:31:45.060: INFO: cilium-2lrb9 started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:31:45.060: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:31:45.060: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:31:45.060: INFO: coredns-autoscaler-6658b4bf85-gdgxf started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:45.060: INFO: Container autoscaler ready: true, restart count 0 Nov 23 23:31:45.060: INFO: coredns-5fcc7b6498-ll578 started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:45.060: INFO: Container coredns ready: true, restart count 0 Nov 23 23:31:45.060: INFO: liveness-c01c2deb-84cd-4e87-b47a-aab4f4ea96e9 started at 2022-11-23 23:29:44 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:45.060: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:31:45.060: INFO: pod-configmaps-6cefcfc7-988c-4c36-a7f4-05776840e9d1 started at 2022-11-23 23:31:22 +0000 UTC (0+3 container statuses recorded) Nov 23 23:31:45.060: INFO: Container createcm-volume-test ready: true, restart count 0 Nov 23 23:31:45.060: INFO: Container delcm-volume-test ready: true, restart count 0 Nov 23 23:31:45.060: INFO: Container updcm-volume-test ready: true, restart count 0 Nov 23 23:31:45.060: INFO: startup-b399f159-4872-469e-94c2-952dd0185871 started at 2022-11-23 23:29:22 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:45.060: INFO: Container busybox ready: false, restart count 0 Nov 23 23:31:45.060: INFO: netserver-2 started at 2022-11-23 23:31:44 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:45.060: INFO: Container webserver ready: false, restart count 0 Nov 23 23:31:45.604: INFO: Latency metrics for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:31:45.604: INFO: Logging node info for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:31:45.758: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-63-80.ap-northeast-1.compute.internal 1d4cdbf2-ca1f-431f-b562-a0bd0f489514 22811 0 2022-11-23 23:18:44 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-63-80.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-63-80.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-3666":"csi-mock-csi-mock-volumes-3666","ebs.csi.aws.com":"i-081735fbd2bce39f2"} io.cilium.network.ipv4-cilium-host:172.20.50.140 io.cilium.network.ipv4-health-ip:172.20.42.94 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:19:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:19:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2022-11-23 23:31:44 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 23:31:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-081735fbd2bce39f2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:19:12 +0000 UTC,LastTransitionTime:2022-11-23 23:19:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:44 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:44 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:44 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:31:44 +0000 UTC,LastTransitionTime:2022-11-23 23:19:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.63.80,},NodeAddress{Type:ExternalIP,Address:13.112.241.172,},NodeAddress{Type:Hostname,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-112-241-172.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29233394293c3c1bf3f4359c382b54,SystemUUID:ec292333-9429-3c3c-1bf3-f4359c382b54,BootID:05304de1-80b3-439c-9eeb-74d1d733f407,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-3666^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-3666^4,DevicePath:,},},Config:nil,},} Nov 23 23:31:45.759: INFO: Logging kubelet events for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:31:45.916: INFO: Logging pods the kubelet thinks is on node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:31:46.085: INFO: pvc-volume-tester-gg6zq started at 2022-11-23 23:31:43 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:46.085: INFO: Container volume-tester ready: false, restart count 0 Nov 23 23:31:46.085: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:31:37 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:46.085: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:31:46.085: INFO: netserver-3 started at 2022-11-23 23:31:41 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:46.085: INFO: Container webserver ready: false, restart count 0 Nov 23 23:31:46.085: INFO: netserver-3 started at 2022-11-23 23:31:44 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:46.085: INFO: Container webserver ready: false, restart count 0 Nov 23 23:31:46.085: INFO: csi-mockplugin-0 started at 2022-11-23 23:31:36 +0000 UTC (0+3 container statuses recorded) Nov 23 23:31:46.085: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:31:46.085: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:31:46.085: INFO: Container mock ready: true, restart count 0 Nov 23 23:31:46.085: INFO: cilium-dgm5r started at 2022-11-23 23:18:44 +0000 UTC (1+1 container statuses recorded) Nov 23 23:31:46.085: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:31:46.085: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:31:46.085: INFO: svc-latency-rc-k9t26 started at 2022-11-23 23:31:23 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:46.085: INFO: Container svc-latency-rc ready: true, restart count 0 Nov 23 23:31:46.085: INFO: test-webserver-21ab2936-d38b-440a-98d8-5b0113a1ffbc started at 2022-11-23 23:31:23 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:46.085: INFO: Container test-webserver ready: true, restart count 0 Nov 23 23:31:46.085: INFO: hostexec-ip-172-20-63-80.ap-northeast-1.compute.internal-vn5mz started at 2022-11-23 23:31:44 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:46.085: INFO: Container agnhost-container ready: false, restart count 0 Nov 23 23:31:46.085: INFO: ebs-csi-node-hlgh2 started at 2022-11-23 23:18:44 +0000 UTC (0+3 container statuses recorded) Nov 23 23:31:46.085: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:31:46.085: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:31:46.085: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:31:46.085: INFO: liveness-183c7acb-0cfd-4281-8b21-0f7d3cc95769 started at 2022-11-23 23:31:14 +0000 UTC (0+1 container statuses recorded) Nov 23 23:31:46.085: INFO: Container agnhost-container ready: true, restart count 1 Nov 23 23:31:46.888: INFO: Latency metrics for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:31:46.888: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "fsgroupchangepolicy-8255" for this suite.
Find pod-d1c4acff-661d-4cbd-9ca6-1bf2d835eb6a mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(OnRootMismatch\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\snew\spod\sfsgroup\sapplied\sto\svolume\scontents$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Nov 23 23:30:02.958: Unexpected error: <*errors.errorString | 0xc0000d0e50>: { s: "pod \"pod-bc795cd7-1fce-41f0-ba6e-72930e9f3766\" is not Running: timed out waiting for the condition", } pod "pod-bc795cd7-1fce-41f0-ba6e-72930e9f3766" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250from junit_02.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Nov 23 23:25:01.098: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename fsgroupchangepolicy �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Nov 23 23:25:01.874: INFO: Creating resource for dynamic PV Nov 23 23:25:01.874: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass fsgroupchangepolicy-3710-e2e-sc9jlxn �[1mSTEP�[0m: creating a claim Nov 23 23:25:02.029: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-3710 with fsgroup 1000 Nov 23 23:30:02.958: FAIL: Unexpected error: <*errors.errorString | 0xc0000d0e50>: { s: "pod \"pod-bc795cd7-1fce-41f0-ba6e-72930e9f3766\" is not Running: timed out waiting for the condition", } pod "pod-bc795cd7-1fce-41f0-ba6e-72930e9f3766" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc0031ee580, 0xc003e3f0d8, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250 +0x1c5 k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:221 +0x219 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000ca5080) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000ca5080) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000ca5080, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 �[1mSTEP�[0m: Deleting pvc Nov 23 23:30:03.112: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.com7bqtm" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "fsgroupchangepolicy-3710". �[1mSTEP�[0m: Found 6 events. Nov 23 23:30:03.578: INFO: At 2022-11-23 23:25:02 +0000 UTC - event for ebs.csi.aws.com7bqtm: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 23:30:03.579: INFO: At 2022-11-23 23:25:02 +0000 UTC - event for ebs.csi.aws.com7bqtm: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 23:30:03.579: INFO: At 2022-11-23 23:25:02 +0000 UTC - event for ebs.csi.aws.com7bqtm: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-3710/ebs.csi.aws.com7bqtm" Nov 23 23:30:03.579: INFO: At 2022-11-23 23:25:12 +0000 UTC - event for ebs.csi.aws.com7bqtm: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-3710-e2e-sc9jlxn": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 23:30:03.579: INFO: At 2022-11-23 23:26:33 +0000 UTC - event for ebs.csi.aws.com7bqtm: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-3710-e2e-sc9jlxn": rpc error: code = Internal desc = Could not create volume "pvc-8f0fa5de-264e-456d-9fff-a28f630669b6": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Nov 23 23:30:03.579: INFO: At 2022-11-23 23:30:03 +0000 UTC - event for pod-bc795cd7-1fce-41f0-ba6e-72930e9f3766: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-3710/ebs.csi.aws.com7bqtm" Nov 23 23:30:03.747: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 23:30:03.747: INFO: pod-bc795cd7-1fce-41f0-ba6e-72930e9f3766 Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2022-11-23 23:30:03 +0000 UTC SchedulerError running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-3710/ebs.csi.aws.com7bqtm"}] Nov 23 23:30:03.747: INFO: Nov 23 23:30:03.904: INFO: Logging node info for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:30:04.057: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-139.ap-northeast-1.compute.internal e52821e5-e82f-478b-be4f-6a60ba5816ce 17680 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-139.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-4383 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-35-139.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-4285":"csi-mock-csi-mock-volumes-4285","csi-mock-csi-mock-volumes-4950":"csi-mock-csi-mock-volumes-4950","ebs.csi.aws.com":"i-0af5c992eacc11f5f"} io.cilium.network.ipv4-cilium-host:172.20.62.73 io.cilium.network.ipv4-health-ip:172.20.63.125 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-11-23 23:27:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:27:56 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2022-11-23 23:28:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0af5c992eacc11f5f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:51 +0000 UTC,LastTransitionTime:2022-11-23 23:18:51 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:30:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:30:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:30:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:30:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.139,},NodeAddress{Type:ExternalIP,Address:13.231.193.129,},NodeAddress{Type:Hostname,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-231-193-129.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec218b5e615d578e61275cbc46e5a775,SystemUUID:ec218b5e-615d-578e-6127-5cbc46e5a775,BootID:15a3520e-c72a-46fe-a3d6-be039aa283b7,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-4285^4 kubernetes.io/csi/csi-mock-csi-mock-volumes-4950^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-4950^4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-4285^4,DevicePath:,},},Config:nil,},} Nov 23 23:30:04.058: INFO: Logging kubelet events for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:30:04.214: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:30:04.378: INFO: csi-mockplugin-0 started at 2022-11-23 23:29:49 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:04.378: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:30:04.378: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:30:04.378: INFO: Container mock ready: true, restart count 0 Nov 23 23:30:04.378: INFO: pvc-volume-tester-wnxb2 started at 2022-11-23 23:30:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:04.378: INFO: Container volume-tester ready: false, restart count 0 Nov 23 23:30:04.378: INFO: csi-mockplugin-0 started at 2022-11-23 23:27:48 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:04.378: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:30:04.378: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:30:04.378: INFO: Container mock ready: true, restart count 0 Nov 23 23:30:04.378: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:27:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:04.378: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:30:04.378: INFO: pvc-volume-tester-cjrdx started at 2022-11-23 23:27:55 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:04.378: INFO: Container volume-tester ready: true, restart count 0 Nov 23 23:30:04.378: INFO: cilium-dw7qk started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:30:04.378: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:30:04.378: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:30:04.378: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:29:50 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:04.378: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:30:04.378: INFO: ebs-csi-node-vbdc7 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:04.378: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:30:04.378: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:30:04.378: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:30:04.930: INFO: Latency metrics for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:30:04.930: INFO: Logging node info for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:30:05.084: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-241.ap-northeast-1.compute.internal 86ed4d71-4560-48af-9e88-d8d85c882f8b 16487 0 2022-11-23 23:16:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-241.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-027602f60f12db3f0"} io.cilium.network.ipv4-cilium-host:172.20.42.20 io.cilium.network.ipv4-health-ip:172.20.32.68 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 23:16:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-11-23 23:16:55 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-11-23 23:17:04 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:17:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 23:18:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:18:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-027602f60f12db3f0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892301824 0} {<nil>} 3801076Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787444224 0} {<nil>} 3698676Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:42 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:17:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.241,},NodeAddress{Type:ExternalIP,Address:54.238.84.104,},NodeAddress{Type:Hostname,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-84-104.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2adfc78fc274df02e3c085caa34395,SystemUUID:ec2adfc7-8fc2-74df-02e3-c085caa34395,BootID:e3997507-167a-4ec7-a6ec-7f97c5d886f5,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42904862,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41478108,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955301,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:30:05.084: INFO: Logging kubelet events for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:30:05.240: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:30:05.399: INFO: kube-apiserver-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+2 container statuses recorded) Nov 23 23:30:05.399: INFO: Container healthcheck ready: true, restart count 0 Nov 23 23:30:05.399: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 23:30:05.399: INFO: ebs-csi-node-94jwj started at 2022-11-23 23:17:02 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:05.399: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:30:05.399: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:30:05.399: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:30:05.399: INFO: kops-controller-fznlr started at 2022-11-23 23:17:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:05.399: INFO: Container kops-controller ready: true, restart count 0 Nov 23 23:30:05.399: INFO: dns-controller-845449865f-zwg9n started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:05.399: INFO: Container dns-controller ready: true, restart count 0 Nov 23 23:30:05.399: INFO: kube-controller-manager-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:05.399: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 23:30:05.399: INFO: kube-scheduler-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:05.399: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 23:30:05.399: INFO: etcd-manager-main-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:05.399: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:30:05.399: INFO: ebs-csi-controller-764b64d679-r8ttz started at 2022-11-23 23:17:03 +0000 UTC (0+5 container statuses recorded) Nov 23 23:30:05.399: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:30:05.399: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:30:05.399: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:30:05.399: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:30:05.399: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:30:05.399: INFO: etcd-manager-events-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:05.399: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:30:05.399: INFO: cilium-qvdj2 started at 2022-11-23 23:17:02 +0000 UTC (1+1 container statuses recorded) Nov 23 23:30:05.399: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:30:05.399: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 23:30:05.399: INFO: cilium-operator-587dd66f87-j2km7 started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:05.399: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 23:30:05.940: INFO: Latency metrics for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:30:05.940: INFO: Logging node info for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:30:06.094: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-42-64.ap-northeast-1.compute.internal 45916c04-689e-47fa-b6fb-bdbad0784fcd 17531 0 2022-11-23 23:18:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-42-64.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-42-64.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-6116":"csi-mock-csi-mock-volumes-6116","csi-mock-csi-mock-volumes-8059":"csi-mock-csi-mock-volumes-8059","ebs.csi.aws.com":"i-049ec10a59a1d20e4"} io.cilium.network.ipv4-cilium-host:172.20.51.104 io.cilium.network.ipv4-health-ip:172.20.34.63 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:29:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-049ec10a59a1d20e4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:56 +0000 UTC,LastTransitionTime:2022-11-23 23:18:56 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:29:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.64,},NodeAddress{Type:ExternalIP,Address:54.238.96.164,},NodeAddress{Type:Hostname,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-96-164.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25c6eb725883a2c2916889bd12be88,SystemUUID:ec25c6eb-7258-83a2-c291-6889bd12be88,BootID:8214bf1e-ee6e-4136-8902-31d30749be64,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:30:06.095: INFO: Logging kubelet events for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:30:06.254: INFO: Logging pods the kubelet thinks is on node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:30:06.416: INFO: pod2 started at 2022-11-23 23:29:53 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:06.416: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:30:06.416: INFO: csi-mockplugin-resizer-0 started at 2022-11-23 23:29:54 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:06.416: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:30:06.416: INFO: cilium-jjmjs started at 2022-11-23 23:18:29 +0000 UTC (1+1 container statuses recorded) Nov 23 23:30:06.416: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:30:06.416: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:30:06.416: INFO: csi-mockplugin-0 started at 2022-11-23 23:29:54 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:06.416: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:30:06.416: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:30:06.416: INFO: Container mock ready: true, restart count 0 Nov 23 23:30:06.416: INFO: hostexec-ip-172-20-42-64.ap-northeast-1.compute.internal-h8h8g started at 2022-11-23 23:29:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:06.416: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:30:06.416: INFO: pvc-volume-tester-bwrp2 started at 2022-11-23 23:30:00 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:06.416: INFO: Container volume-tester ready: true, restart count 0 Nov 23 23:30:06.416: INFO: local-injector started at 2022-11-23 23:30:04 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:06.416: INFO: Container local-injector ready: false, restart count 0 Nov 23 23:30:06.416: INFO: pod-subpath-test-preprovisionedpv-wncs started at 2022-11-23 23:30:05 +0000 UTC (2+2 container statuses recorded) Nov 23 23:30:06.416: INFO: Init container init-volume-preprovisionedpv-wncs ready: false, restart count 0 Nov 23 23:30:06.416: INFO: Init container test-init-subpath-preprovisionedpv-wncs ready: false, restart count 0 Nov 23 23:30:06.416: INFO: Container test-container-subpath-preprovisionedpv-wncs ready: false, restart count 0 Nov 23 23:30:06.416: INFO: Container test-container-volume-preprovisionedpv-wncs ready: false, restart count 0 Nov 23 23:30:06.416: INFO: coredns-5fcc7b6498-s4jjd started at 2022-11-23 23:19:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:06.416: INFO: Container coredns ready: true, restart count 0 Nov 23 23:30:06.416: INFO: hostexec-ip-172-20-42-64.ap-northeast-1.compute.internal-2zdds started at 2022-11-23 23:29:55 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:06.416: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:30:06.416: INFO: csi-mockplugin-0 started at 2022-11-23 23:29:18 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:06.416: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:30:06.416: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:30:06.416: INFO: Container mock ready: true, restart count 0 Nov 23 23:30:06.416: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:29:18 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:06.416: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:30:06.416: INFO: ebs-csi-node-fq5pw started at 2022-11-23 23:18:29 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:06.416: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:30:06.416: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:30:06.416: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:30:06.416: INFO: inline-volume-xhkps started at 2022-11-23 23:28:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:06.416: INFO: Container volume-tester ready: false, restart count 0 Nov 23 23:30:06.978: INFO: Latency metrics for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:30:06.978: INFO: Logging node info for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:30:07.132: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-200.ap-northeast-1.compute.internal 8f32a357-4fd8-4fcd-ab21-8b69dbd931d6 13785 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-200.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-6944":"csi-mock-csi-mock-volumes-6944","ebs.csi.aws.com":"i-0d4c764fea6a24b23"} io.cilium.network.ipv4-cilium-host:172.20.61.17 io.cilium.network.ipv4-health-ip:172.20.60.108 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:50 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:24:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0d4c764fea6a24b23,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:50 +0000 UTC,LastTransitionTime:2022-11-23 23:18:50 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.200,},NodeAddress{Type:ExternalIP,Address:3.113.30.221,},NodeAddress{Type:Hostname,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-113-30-221.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ef1c5e899b01a14fef22e61702fe5,SystemUUID:ec2ef1c5-e899-b01a-14fe-f22e61702fe5,BootID:021f6d58-0800-4a46-b65e-3513c48dff08,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:30:07.133: INFO: Logging kubelet events for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:30:07.289: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:30:07.447: INFO: ebs-csi-node-dl575 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:07.447: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:30:07.447: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:30:07.447: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:30:07.447: INFO: csi-mockplugin-0 started at 2022-11-23 23:27:49 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:07.447: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:30:07.447: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:30:07.447: INFO: Container mock ready: true, restart count 0 Nov 23 23:30:07.447: INFO: startup-b399f159-4872-469e-94c2-952dd0185871 started at 2022-11-23 23:29:22 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:07.447: INFO: Container busybox ready: false, restart count 0 Nov 23 23:30:07.447: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:27:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:07.447: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:30:07.447: INFO: execpodvsgzs started at 2022-11-23 23:29:59 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:07.447: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:30:07.447: INFO: cilium-2lrb9 started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:30:07.447: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:30:07.447: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:30:07.447: INFO: coredns-autoscaler-6658b4bf85-gdgxf started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:07.447: INFO: Container autoscaler ready: true, restart count 0 Nov 23 23:30:07.447: INFO: coredns-5fcc7b6498-ll578 started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:07.447: INFO: Container coredns ready: true, restart count 0 Nov 23 23:30:07.447: INFO: liveness-c01c2deb-84cd-4e87-b47a-aab4f4ea96e9 started at 2022-11-23 23:29:44 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:07.447: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:30:08.004: INFO: Latency metrics for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:30:08.005: INFO: Logging node info for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:30:08.158: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-63-80.ap-northeast-1.compute.internal 1d4cdbf2-ca1f-431f-b562-a0bd0f489514 17716 0 2022-11-23 23:18:44 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-63-80.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-63-80.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-3485":"ip-172-20-63-80.ap-northeast-1.compute.internal","ebs.csi.aws.com":"i-081735fbd2bce39f2"} io.cilium.network.ipv4-cilium-host:172.20.50.140 io.cilium.network.ipv4-health-ip:172.20.42.94 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:19:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:19:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:30:04 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-081735fbd2bce39f2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:19:12 +0000 UTC,LastTransitionTime:2022-11-23 23:19:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:30:04 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:30:04 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:30:04 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:30:04 +0000 UTC,LastTransitionTime:2022-11-23 23:19:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.63.80,},NodeAddress{Type:ExternalIP,Address:13.112.241.172,},NodeAddress{Type:Hostname,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-112-241-172.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29233394293c3c1bf3f4359c382b54,SystemUUID:ec292333-9429-3c3c-1bf3-f4359c382b54,BootID:05304de1-80b3-439c-9eeb-74d1d733f407,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:30:08.159: INFO: Logging kubelet events for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:30:08.314: INFO: Logging pods the kubelet thinks is on node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:30:08.472: INFO: ebs-csi-node-hlgh2 started at 2022-11-23 23:18:44 +0000 UTC (0+3 container statuses recorded) Nov 23 23:30:08.472: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:30:08.472: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:30:08.472: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:30:08.472: INFO: test-rs-j4qnb started at 2022-11-23 23:30:07 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:08.472: INFO: Container httpd ready: false, restart count 0 Nov 23 23:30:08.472: INFO: pod1 started at 2022-11-23 23:29:50 +0000 UTC (0+1 container statuses recorded) Nov 23 23:30:08.472: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:30:08.472: INFO: cilium-dgm5r started at 2022-11-23 23:18:44 +0000 UTC (1+1 container statuses recorded) Nov 23 23:30:08.472: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:30:08.472: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:30:08.472: INFO: csi-hostpathplugin-0 started at 2022-11-23 23:29:15 +0000 UTC (0+7 container statuses recorded) Nov 23 23:30:08.472: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:30:08.472: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:30:08.472: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:30:08.472: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 23 23:30:08.472: INFO: Container hostpath ready: true, restart count 0 Nov 23 23:30:08.472: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:30:08.472: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:30:09.060: INFO: Latency metrics for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:30:09.060: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "fsgroupchangepolicy-3710" for this suite.
Find pod-bc795cd7-1fce-41f0-ba6e-72930e9f3766 mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(OnRootMismatch\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\svolume\scontents\sownership\schanged\sin\sfirst\spod\,\snew\spod\swith\sdifferent\sfsgroup\sapplied\sto\sthe\svolume\scontents$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Nov 23 23:32:48.019: Unexpected error: <*errors.errorString | 0xc003d19450>: { s: "pod \"pod-83dda85c-2c1d-4441-a513-7789b1f1748c\" is not Running: timed out waiting for the condition", } pod "pod-83dda85c-2c1d-4441-a513-7789b1f1748c" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250from junit_04.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Nov 23 23:27:46.159: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename fsgroupchangepolicy �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with different fsgroup applied to the volume contents /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Nov 23 23:27:46.926: INFO: Creating resource for dynamic PV Nov 23 23:27:46.926: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass fsgroupchangepolicy-4779-e2e-scrd2jt �[1mSTEP�[0m: creating a claim Nov 23 23:27:47.081: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-4779 with fsgroup 1000 Nov 23 23:32:48.019: FAIL: Unexpected error: <*errors.errorString | 0xc003d19450>: { s: "pod \"pod-83dda85c-2c1d-4441-a513-7789b1f1748c\" is not Running: timed out waiting for the condition", } pod "pod-83dda85c-2c1d-4441-a513-7789b1f1748c" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc003c9d1e0, 0xc0037fd0d8, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250 +0x1c5 k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:221 +0x219 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc0000f0d80) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc0000f0d80) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc0000f0d80, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 �[1mSTEP�[0m: Deleting pvc Nov 23 23:32:48.174: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.com29jdm" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "fsgroupchangepolicy-4779". �[1mSTEP�[0m: Found 5 events. Nov 23 23:32:48.652: INFO: At 2022-11-23 23:27:47 +0000 UTC - event for ebs.csi.aws.com29jdm: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 23:32:48.652: INFO: At 2022-11-23 23:27:47 +0000 UTC - event for ebs.csi.aws.com29jdm: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 23:32:48.652: INFO: At 2022-11-23 23:27:47 +0000 UTC - event for ebs.csi.aws.com29jdm: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-4779/ebs.csi.aws.com29jdm" Nov 23 23:32:48.652: INFO: At 2022-11-23 23:27:57 +0000 UTC - event for ebs.csi.aws.com29jdm: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-4779-e2e-scrd2jt": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 23:32:48.652: INFO: At 2022-11-23 23:32:48 +0000 UTC - event for pod-83dda85c-2c1d-4441-a513-7789b1f1748c: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-4779/ebs.csi.aws.com29jdm" Nov 23 23:32:48.807: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 23:32:48.807: INFO: pod-83dda85c-2c1d-4441-a513-7789b1f1748c Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2022-11-23 23:32:48 +0000 UTC SchedulerError running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-4779/ebs.csi.aws.com29jdm"}] Nov 23 23:32:48.807: INFO: Nov 23 23:32:48.962: INFO: Logging node info for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:32:49.117: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-139.ap-northeast-1.compute.internal e52821e5-e82f-478b-be4f-6a60ba5816ce 21725 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-139.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-4383 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-35-139.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0af5c992eacc11f5f"} io.cilium.network.ipv4-cilium-host:172.20.62.73 io.cilium.network.ipv4-health-ip:172.20.63.125 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-11-23 23:27:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:mounted_volume_expand":{}}}} } {kubelet Update v1 2022-11-23 23:31:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0af5c992eacc11f5f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:51 +0000 UTC,LastTransitionTime:2022-11-23 23:18:51 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:31:13 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.139,},NodeAddress{Type:ExternalIP,Address:13.231.193.129,},NodeAddress{Type:Hostname,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-231-193-129.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec218b5e615d578e61275cbc46e5a775,SystemUUID:ec218b5e-615d-578e-6127-5cbc46e5a775,BootID:15a3520e-c72a-46fe-a3d6-be039aa283b7,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:32:49.118: INFO: Logging kubelet events for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:32:49.275: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:32:49.591: INFO: netserver-0 started at 2022-11-23 23:32:16 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:49.591: INFO: Container webserver ready: true, restart count 0 Nov 23 23:32:49.591: INFO: ss2-0 started at 2022-11-23 23:32:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:49.591: INFO: Container webserver ready: false, restart count 0 Nov 23 23:32:49.591: INFO: csi-mockplugin-0 started at 2022-11-23 23:32:49 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:49.591: INFO: Container csi-provisioner ready: false, restart count 0 Nov 23 23:32:49.591: INFO: Container driver-registrar ready: false, restart count 0 Nov 23 23:32:49.591: INFO: Container mock ready: false, restart count 0 Nov 23 23:32:49.591: INFO: netserver-0 started at 2022-11-23 23:32:33 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:49.591: INFO: Container webserver ready: true, restart count 0 Nov 23 23:32:49.591: INFO: affinity-nodeport-transition-z5dbd started at 2022-11-23 23:32:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:49.591: INFO: Container affinity-nodeport-transition ready: true, restart count 0 Nov 23 23:32:49.591: INFO: test-container-pod started at 2022-11-23 23:32:40 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:49.591: INFO: Container webserver ready: true, restart count 0 Nov 23 23:32:49.591: INFO: cilium-dw7qk started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:32:49.591: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:32:49.591: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:32:49.591: INFO: verify-service-up-host-exec-pod started at 2022-11-23 23:32:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:49.591: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:32:49.591: INFO: ebs-csi-node-vbdc7 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:49.591: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:49.591: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:49.591: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:49.591: INFO: service-headless-qv59x started at 2022-11-23 23:32:40 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:49.591: INFO: Container service-headless ready: true, restart count 0 Nov 23 23:32:49.591: INFO: service-headless-toggled-lbcbn started at 2022-11-23 23:32:43 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:49.591: INFO: Container service-headless-toggled ready: true, restart count 0 Nov 23 23:32:50.153: INFO: Latency metrics for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:32:50.154: INFO: Logging node info for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:32:50.308: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-241.ap-northeast-1.compute.internal 86ed4d71-4560-48af-9e88-d8d85c882f8b 16487 0 2022-11-23 23:16:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-241.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-027602f60f12db3f0"} io.cilium.network.ipv4-cilium-host:172.20.42.20 io.cilium.network.ipv4-health-ip:172.20.32.68 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 23:16:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-11-23 23:16:55 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-11-23 23:17:04 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:17:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 23:18:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:18:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-027602f60f12db3f0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892301824 0} {<nil>} 3801076Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787444224 0} {<nil>} 3698676Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:42 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:29:21 +0000 UTC,LastTransitionTime:2022-11-23 23:17:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.241,},NodeAddress{Type:ExternalIP,Address:54.238.84.104,},NodeAddress{Type:Hostname,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-84-104.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2adfc78fc274df02e3c085caa34395,SystemUUID:ec2adfc7-8fc2-74df-02e3-c085caa34395,BootID:e3997507-167a-4ec7-a6ec-7f97c5d886f5,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42904862,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41478108,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955301,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:32:50.308: INFO: Logging kubelet events for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:32:50.465: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:32:50.625: INFO: etcd-manager-events-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:50.625: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:32:50.625: INFO: cilium-qvdj2 started at 2022-11-23 23:17:02 +0000 UTC (1+1 container statuses recorded) Nov 23 23:32:50.625: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:32:50.625: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 23:32:50.625: INFO: cilium-operator-587dd66f87-j2km7 started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:50.625: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 23:32:50.625: INFO: ebs-csi-controller-764b64d679-r8ttz started at 2022-11-23 23:17:03 +0000 UTC (0+5 container statuses recorded) Nov 23 23:32:50.625: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:32:50.625: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:32:50.625: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:32:50.625: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:50.625: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:50.625: INFO: ebs-csi-node-94jwj started at 2022-11-23 23:17:02 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:50.625: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:50.625: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:50.625: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:50.625: INFO: kops-controller-fznlr started at 2022-11-23 23:17:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:50.625: INFO: Container kops-controller ready: true, restart count 0 Nov 23 23:32:50.625: INFO: dns-controller-845449865f-zwg9n started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:50.625: INFO: Container dns-controller ready: true, restart count 0 Nov 23 23:32:50.625: INFO: kube-controller-manager-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:50.625: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 23:32:50.625: INFO: kube-scheduler-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:50.625: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 23:32:50.625: INFO: etcd-manager-main-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:50.625: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:32:50.625: INFO: kube-apiserver-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+2 container statuses recorded) Nov 23 23:32:50.625: INFO: Container healthcheck ready: true, restart count 0 Nov 23 23:32:50.625: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 23:32:51.210: INFO: Latency metrics for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:32:51.210: INFO: Logging node info for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:32:51.366: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-42-64.ap-northeast-1.compute.internal 45916c04-689e-47fa-b6fb-bdbad0784fcd 24098 0 2022-11-23 23:18:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-42-64.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-42-64.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-049ec10a59a1d20e4"} io.cilium.network.ipv4-cilium-host:172.20.51.104 io.cilium.network.ipv4-health-ip:172.20.34.63 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:32:09 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-049ec10a59a1d20e4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:56 +0000 UTC,LastTransitionTime:2022-11-23 23:18:56 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:32:09 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:32:09 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:32:09 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:32:09 +0000 UTC,LastTransitionTime:2022-11-23 23:18:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.64,},NodeAddress{Type:ExternalIP,Address:54.238.96.164,},NodeAddress{Type:Hostname,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-96-164.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25c6eb725883a2c2916889bd12be88,SystemUUID:ec25c6eb-7258-83a2-c291-6889bd12be88,BootID:8214bf1e-ee6e-4136-8902-31d30749be64,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:32:51.367: INFO: Logging kubelet events for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:32:51.531: INFO: Logging pods the kubelet thinks is on node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:32:51.695: INFO: ebs-csi-node-fq5pw started at 2022-11-23 23:18:29 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:51.695: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:51.695: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:51.695: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:51.695: INFO: affinity-nodeport-transition-9877k started at 2022-11-23 23:32:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:51.695: INFO: Container affinity-nodeport-transition ready: true, restart count 0 Nov 23 23:32:51.695: INFO: netserver-1 started at 2022-11-23 23:32:16 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:51.695: INFO: Container webserver ready: true, restart count 0 Nov 23 23:32:51.695: INFO: service-headless-toggled-9cnb8 started at 2022-11-23 23:32:43 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:51.695: INFO: Container service-headless-toggled ready: true, restart count 0 Nov 23 23:32:51.695: INFO: adopt-release-pmrnz started at 2022-11-23 23:32:51 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:51.695: INFO: Container c ready: false, restart count 0 Nov 23 23:32:51.695: INFO: pfpod started at 2022-11-23 23:32:38 +0000 UTC (0+2 container statuses recorded) Nov 23 23:32:51.695: INFO: Container portforwardtester ready: false, restart count 0 Nov 23 23:32:51.695: INFO: Container readiness ready: false, restart count 0 Nov 23 23:32:51.695: INFO: netserver-1 started at 2022-11-23 23:32:33 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:51.695: INFO: Container webserver ready: false, restart count 0 Nov 23 23:32:51.695: INFO: cilium-jjmjs started at 2022-11-23 23:18:29 +0000 UTC (1+1 container statuses recorded) Nov 23 23:32:51.695: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:32:51.695: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:32:51.695: INFO: service-headless-5qdww started at 2022-11-23 23:32:40 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:51.695: INFO: Container service-headless ready: true, restart count 0 Nov 23 23:32:51.695: INFO: ss2-1 started at 2022-11-23 23:32:50 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:51.695: INFO: Container webserver ready: false, restart count 0 Nov 23 23:32:51.695: INFO: condition-test-csn5v started at 2022-11-23 23:32:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:51.695: INFO: Container httpd ready: false, restart count 0 Nov 23 23:32:51.695: INFO: coredns-5fcc7b6498-s4jjd started at 2022-11-23 23:19:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:51.695: INFO: Container coredns ready: true, restart count 0 Nov 23 23:32:51.695: INFO: pod-ready started at 2022-11-23 23:32:24 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:51.695: INFO: Container pod-readiness-gate ready: true, restart count 0 Nov 23 23:32:52.279: INFO: Latency metrics for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:32:52.279: INFO: Logging node info for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:32:52.434: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-200.ap-northeast-1.compute.internal 8f32a357-4fd8-4fcd-ab21-8b69dbd931d6 23091 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-200.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9079":"csi-mock-csi-mock-volumes-9079","ebs.csi.aws.com":"i-0d4c764fea6a24b23"} io.cilium.network.ipv4-cilium-host:172.20.61.17 io.cilium.network.ipv4-health-ip:172.20.60.108 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:50 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:24:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0d4c764fea6a24b23,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:50 +0000 UTC,LastTransitionTime:2022-11-23 23:18:50 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:27:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.200,},NodeAddress{Type:ExternalIP,Address:3.113.30.221,},NodeAddress{Type:Hostname,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-113-30-221.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ef1c5e899b01a14fef22e61702fe5,SystemUUID:ec2ef1c5-e899-b01a-14fe-f22e61702fe5,BootID:021f6d58-0800-4a46-b65e-3513c48dff08,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:32:52.435: INFO: Logging kubelet events for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:32:52.591: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:32:52.751: INFO: cilium-2lrb9 started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:32:52.752: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:32:52.752: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:32:52.752: INFO: startup-b399f159-4872-469e-94c2-952dd0185871 started at 2022-11-23 23:29:22 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:52.752: INFO: Container busybox ready: false, restart count 0 Nov 23 23:32:52.752: INFO: verify-service-up-exec-pod-kbdhq started at 2022-11-23 23:32:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:52.752: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:32:52.752: INFO: csi-mockplugin-0 started at 2022-11-23 23:31:52 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:52.752: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:32:52.752: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:32:52.752: INFO: Container mock ready: true, restart count 0 Nov 23 23:32:52.752: INFO: adopt-release-2tz9b started at 2022-11-23 23:32:51 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:52.752: INFO: Container c ready: false, restart count 0 Nov 23 23:32:52.752: INFO: ebs-csi-node-dl575 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:52.752: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:52.752: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:52.752: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:52.752: INFO: service-headless-jdmsc started at 2022-11-23 23:32:40 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:52.752: INFO: Container service-headless ready: true, restart count 0 Nov 23 23:32:52.752: INFO: ss2-2 started at 2022-11-23 23:32:52 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:52.752: INFO: Container webserver ready: false, restart count 0 Nov 23 23:32:52.752: INFO: coredns-autoscaler-6658b4bf85-gdgxf started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:52.752: INFO: Container autoscaler ready: true, restart count 0 Nov 23 23:32:52.752: INFO: coredns-5fcc7b6498-ll578 started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:52.752: INFO: Container coredns ready: true, restart count 0 Nov 23 23:32:52.752: INFO: liveness-c01c2deb-84cd-4e87-b47a-aab4f4ea96e9 started at 2022-11-23 23:29:44 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:52.752: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:32:52.752: INFO: affinity-nodeport-transition-fnxdp started at 2022-11-23 23:32:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:52.752: INFO: Container affinity-nodeport-transition ready: true, restart count 0 Nov 23 23:32:52.752: INFO: pvc-volume-tester-qtr6f started at 2022-11-23 23:32:04 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:52.752: INFO: Container volume-tester ready: true, restart count 0 Nov 23 23:32:52.752: INFO: netserver-2 started at 2022-11-23 23:32:34 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:52.752: INFO: Container webserver ready: false, restart count 0 Nov 23 23:32:52.752: INFO: condition-test-2qhkc started at 2022-11-23 23:32:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:52.752: INFO: Container httpd ready: true, restart count 0 Nov 23 23:32:53.318: INFO: Latency metrics for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:32:53.318: INFO: Logging node info for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:32:53.472: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-63-80.ap-northeast-1.compute.internal 1d4cdbf2-ca1f-431f-b562-a0bd0f489514 25203 0 2022-11-23 23:18:44 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-63-80.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-63-80.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-volume-expand-4758":"ip-172-20-63-80.ap-northeast-1.compute.internal","ebs.csi.aws.com":"i-081735fbd2bce39f2"} io.cilium.network.ipv4-cilium-host:172.20.50.140 io.cilium.network.ipv4-health-ip:172.20.42.94 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:19:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:19:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:31:54 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {kube-controller-manager Update v1 2022-11-23 23:32:50 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-081735fbd2bce39f2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:19:12 +0000 UTC,LastTransitionTime:2022-11-23 23:19:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:32:45 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:32:45 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:32:45 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:32:45 +0000 UTC,LastTransitionTime:2022-11-23 23:19:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.63.80,},NodeAddress{Type:ExternalIP,Address:13.112.241.172,},NodeAddress{Type:Hostname,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-112-241-172.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29233394293c3c1bf3f4359c382b54,SystemUUID:ec292333-9429-3c3c-1bf3-f4359c382b54,BootID:05304de1-80b3-439c-9eeb-74d1d733f407,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-expand-4758^2373b3fe-6b87-11ed-9d92-2a32a0515425,DevicePath:,},},Config:nil,},} Nov 23 23:32:53.473: INFO: Logging kubelet events for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:32:53.629: INFO: Logging pods the kubelet thinks is on node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:32:53.802: INFO: csi-hostpathplugin-0 started at 2022-11-23 23:32:44 +0000 UTC (0+7 container statuses recorded) Nov 23 23:32:53.803: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:32:53.803: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:32:53.803: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:32:53.803: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 23 23:32:53.803: INFO: Container hostpath ready: true, restart count 0 Nov 23 23:32:53.803: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:53.803: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:53.803: INFO: cilium-dgm5r started at 2022-11-23 23:18:44 +0000 UTC (1+1 container statuses recorded) Nov 23 23:32:53.803: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:32:53.803: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:32:53.803: INFO: netserver-3 started at 2022-11-23 23:32:34 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:53.803: INFO: Container webserver ready: false, restart count 0 Nov 23 23:32:53.803: INFO: test-webserver-21ab2936-d38b-440a-98d8-5b0113a1ffbc started at 2022-11-23 23:31:23 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:53.803: INFO: Container test-webserver ready: true, restart count 0 Nov 23 23:32:53.803: INFO: ebs-csi-node-hlgh2 started at 2022-11-23 23:18:44 +0000 UTC (0+3 container statuses recorded) Nov 23 23:32:53.803: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:32:53.803: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:32:53.803: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:32:53.803: INFO: service-headless-toggled-ccvjj started at 2022-11-23 23:32:43 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:53.803: INFO: Container service-headless-toggled ready: true, restart count 0 Nov 23 23:32:53.803: INFO: liveness-183c7acb-0cfd-4281-8b21-0f7d3cc95769 started at 2022-11-23 23:31:14 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:53.803: INFO: Container agnhost-container ready: true, restart count 4 Nov 23 23:32:53.803: INFO: pod-bd9e1ee7-dd99-4059-8cc4-bf98138007a2 started at 2022-11-23 23:32:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:32:53.803: INFO: Container write-pod ready: false, restart count 0 Nov 23 23:32:54.351: INFO: Latency metrics for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:32:54.351: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "fsgroupchangepolicy-4779" for this suite.
Find pod-83dda85c-2c1d-4441-a513-7789b1f1748c mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sfsgroupchangepolicy\s\(OnRootMismatch\)\[LinuxOnly\]\,\spod\screated\swith\san\sinitial\sfsgroup\,\svolume\scontents\sownership\schanged\sin\sfirst\spod\,\snew\spod\swith\ssame\sfsgroup\sskips\sownership\schanges\sto\sthe\svolume\scontents$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Nov 23 23:52:45.714: Unexpected error: <*errors.errorString | 0xc004d269a0>: { s: "pod \"pod-feb6db0b-4eb5-47f0-bdb8-f95072e56e8c\" is not Running: timed out waiting for the condition", } pod "pod-feb6db0b-4eb5-47f0-bdb8-f95072e56e8c" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250from junit_06.xml
[BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Nov 23 23:47:43.848: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename fsgroupchangepolicy �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup skips ownership changes to the volume contents /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208 Nov 23 23:47:44.622: INFO: Creating resource for dynamic PV Nov 23 23:47:44.622: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} �[1mSTEP�[0m: creating a StorageClass fsgroupchangepolicy-7896-e2e-scn7qrx �[1mSTEP�[0m: creating a claim Nov 23 23:47:44.778: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil �[1mSTEP�[0m: Creating Pod in namespace fsgroupchangepolicy-7896 with fsgroup 1000 Nov 23 23:52:45.714: FAIL: Unexpected error: <*errors.errorString | 0xc004d269a0>: { s: "pod \"pod-feb6db0b-4eb5-47f0-bdb8-f95072e56e8c\" is not Running: timed out waiting for the condition", } pod "pod-feb6db0b-4eb5-47f0-bdb8-f95072e56e8c" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.createPodAndVerifyContentGid(0xc003d6f1e0, 0xc0050c30d8, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:250 +0x1c5 k8s.io/kubernetes/test/e2e/storage/testsuites.(*fsGroupChangePolicyTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:221 +0x219 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000ccc180) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000ccc180) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000ccc180, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 �[1mSTEP�[0m: Deleting pvc Nov 23 23:52:45.870: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comdbhl5" �[1mSTEP�[0m: Deleting sc [AfterEach] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "fsgroupchangepolicy-7896". �[1mSTEP�[0m: Found 5 events. Nov 23 23:52:46.336: INFO: At 2022-11-23 23:47:44 +0000 UTC - event for ebs.csi.aws.comdbhl5: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 23:52:46.336: INFO: At 2022-11-23 23:47:45 +0000 UTC - event for ebs.csi.aws.comdbhl5: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 23:52:46.336: INFO: At 2022-11-23 23:47:45 +0000 UTC - event for ebs.csi.aws.comdbhl5: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-7896/ebs.csi.aws.comdbhl5" Nov 23 23:52:46.336: INFO: At 2022-11-23 23:47:55 +0000 UTC - event for ebs.csi.aws.comdbhl5: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "fsgroupchangepolicy-7896-e2e-scn7qrx": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 23:52:46.336: INFO: At 2022-11-23 23:52:46 +0000 UTC - event for pod-feb6db0b-4eb5-47f0-bdb8-f95072e56e8c: {default-scheduler } FailedScheduling: running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-7896/ebs.csi.aws.comdbhl5" Nov 23 23:52:46.491: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 23:52:46.492: INFO: pod-feb6db0b-4eb5-47f0-bdb8-f95072e56e8c Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2022-11-23 23:52:46 +0000 UTC SchedulerError running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "fsgroupchangepolicy-7896/ebs.csi.aws.comdbhl5"}] Nov 23 23:52:46.492: INFO: Nov 23 23:52:46.648: INFO: Logging node info for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:52:46.803: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-139.ap-northeast-1.compute.internal e52821e5-e82f-478b-be4f-6a60ba5816ce 46314 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-139.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node mounted_volume_expand:mounted-volume-expand-4383 node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-35-139.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-295":"csi-mock-csi-mock-volumes-295","ebs.csi.aws.com":"i-0af5c992eacc11f5f"} io.cilium.network.ipv4-cilium-host:172.20.62.73 io.cilium.network.ipv4-health-ip:172.20.63.125 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:39:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {e2e.test Update v1 2022-11-23 23:41:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{},"f:mounted_volume_expand":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0af5c992eacc11f5f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:51 +0000 UTC,LastTransitionTime:2022-11-23 23:18:51 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:48:24 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:48:24 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:48:24 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:48:24 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.139,},NodeAddress{Type:ExternalIP,Address:13.231.193.129,},NodeAddress{Type:Hostname,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-231-193-129.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec218b5e615d578e61275cbc46e5a775,SystemUUID:ec218b5e-615d-578e-6127-5cbc46e5a775,BootID:15a3520e-c72a-46fe-a3d6-be039aa283b7,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:52:46.803: INFO: Logging kubelet events for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:52:46.961: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:52:47.121: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-cprng started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:47.122: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-jk7vl started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:47.122: INFO: csi-mockplugin-0 started at 2022-11-23 23:47:13 +0000 UTC (0+3 container statuses recorded) Nov 23 23:52:47.122: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:52:47.122: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:52:47.122: INFO: Container mock ready: true, restart count 0 Nov 23 23:52:47.122: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-l242f started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:47.122: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-fhf6d started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:47.122: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-7dzpn started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:47.122: INFO: local-injector started at 2022-11-23 23:48:19 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container local-injector ready: false, restart count 0 Nov 23 23:52:47.122: INFO: pod-8b76cd25-72f5-4456-938e-67d208d57ec8 started at 2022-11-23 23:50:33 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container test-container ready: false, restart count 0 Nov 23 23:52:47.122: INFO: cilium-dw7qk started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:52:47.122: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:52:47.122: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-w7cmd started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:47.122: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-4b7zt started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:47.122: INFO: ebs-csi-node-vbdc7 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:52:47.122: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:52:47.122: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:52:47.122: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:52:47.122: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-rvnlm started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:47.122: INFO: downward-api-647e0daf-d60c-4568-86aa-1c0b0f010801 started at 2022-11-23 23:48:09 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container dapi-container ready: false, restart count 0 Nov 23 23:52:47.122: INFO: pvc-volume-tester-6jp4n started at 2022-11-23 23:48:25 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container volume-tester ready: false, restart count 0 Nov 23 23:52:47.122: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-hzp2t started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:47.122: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-zz27d started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:47.122: INFO: test-deployment-56c98d85f9-njt7s started at 2022-11-23 23:47:59 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container test-deployment ready: true, restart count 0 Nov 23 23:52:47.122: INFO: hostexec-ip-172-20-35-139.ap-northeast-1.compute.internal-ql7zb started at 2022-11-23 23:48:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:52:47.122: INFO: pod-configmaps-a2e1f4b2-c35b-4814-b30a-2c6103aa6e47 started at 2022-11-23 23:48:07 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container agnhost-container ready: false, restart count 0 Nov 23 23:52:47.122: INFO: client-envvars-4b338700-5b70-4903-92a3-72f3915d804f started at 2022-11-23 23:48:08 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:47.122: INFO: Container env3cont ready: false, restart count 0 Nov 23 23:52:47.689: INFO: Latency metrics for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:52:47.689: INFO: Logging node info for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:52:47.844: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-241.ap-northeast-1.compute.internal 86ed4d71-4560-48af-9e88-d8d85c882f8b 46497 0 2022-11-23 23:16:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-241.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-027602f60f12db3f0"} io.cilium.network.ipv4-cilium-host:172.20.42.20 io.cilium.network.ipv4-health-ip:172.20.32.68 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 23:16:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-11-23 23:16:55 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-11-23 23:17:04 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:17:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 23:18:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:18:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-027602f60f12db3f0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892301824 0} {<nil>} 3801076Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787444224 0} {<nil>} 3698676Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:42 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:49:22 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:49:22 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:49:22 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:49:22 +0000 UTC,LastTransitionTime:2022-11-23 23:17:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.241,},NodeAddress{Type:ExternalIP,Address:54.238.84.104,},NodeAddress{Type:Hostname,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-84-104.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2adfc78fc274df02e3c085caa34395,SystemUUID:ec2adfc7-8fc2-74df-02e3-c085caa34395,BootID:e3997507-167a-4ec7-a6ec-7f97c5d886f5,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42904862,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41478108,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955301,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:52:47.845: INFO: Logging kubelet events for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:52:48.002: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:52:48.163: INFO: etcd-manager-main-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:48.163: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:52:48.163: INFO: kube-apiserver-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+2 container statuses recorded) Nov 23 23:52:48.164: INFO: Container healthcheck ready: true, restart count 0 Nov 23 23:52:48.164: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 23:52:48.164: INFO: ebs-csi-node-94jwj started at 2022-11-23 23:17:02 +0000 UTC (0+3 container statuses recorded) Nov 23 23:52:48.164: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:52:48.164: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:52:48.164: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:52:48.164: INFO: kops-controller-fznlr started at 2022-11-23 23:17:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:48.164: INFO: Container kops-controller ready: true, restart count 0 Nov 23 23:52:48.164: INFO: dns-controller-845449865f-zwg9n started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:48.164: INFO: Container dns-controller ready: true, restart count 0 Nov 23 23:52:48.164: INFO: kube-controller-manager-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:48.164: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 23:52:48.164: INFO: kube-scheduler-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:48.164: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 23:52:48.164: INFO: cilium-operator-587dd66f87-j2km7 started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:48.164: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 23:52:48.164: INFO: ebs-csi-controller-764b64d679-r8ttz started at 2022-11-23 23:17:03 +0000 UTC (0+5 container statuses recorded) Nov 23 23:52:48.164: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:52:48.164: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:52:48.164: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:52:48.164: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:52:48.164: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:52:48.164: INFO: etcd-manager-events-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:48.164: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:52:48.164: INFO: cilium-qvdj2 started at 2022-11-23 23:17:02 +0000 UTC (1+1 container statuses recorded) Nov 23 23:52:48.164: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:52:48.164: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 23:52:48.701: INFO: Latency metrics for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:52:48.701: INFO: Logging node info for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:52:48.856: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-42-64.ap-northeast-1.compute.internal 45916c04-689e-47fa-b6fb-bdbad0784fcd 46671 0 2022-11-23 23:18:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a io.kubernetes.storage.mock/node:some-mock-node kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-42-64.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-42-64.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-049ec10a59a1d20e4"} io.cilium.network.ipv4-cilium-host:172.20.51.104 io.cilium.network.ipv4-health-ip:172.20.34.63 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-11-23 23:41:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2022-11-23 23:49:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-049ec10a59a1d20e4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:56 +0000 UTC,LastTransitionTime:2022-11-23 23:18:56 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:49:39 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:49:39 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:49:39 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:49:39 +0000 UTC,LastTransitionTime:2022-11-23 23:18:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.64,},NodeAddress{Type:ExternalIP,Address:54.238.96.164,},NodeAddress{Type:Hostname,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-96-164.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25c6eb725883a2c2916889bd12be88,SystemUUID:ec25c6eb-7258-83a2-c291-6889bd12be88,BootID:8214bf1e-ee6e-4136-8902-31d30749be64,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:52:48.857: INFO: Logging kubelet events for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:52:49.015: INFO: Logging pods the kubelet thinks is on node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:52:49.181: INFO: coredns-5fcc7b6498-s4jjd started at 2022-11-23 23:19:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:49.181: INFO: Container coredns ready: true, restart count 0 Nov 23 23:52:49.181: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-tjvrw started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:49.181: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:49.181: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-2g6dd started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:49.181: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:49.181: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-mpnx6 started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:49.181: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:49.181: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-c99b4 started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:49.181: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:49.181: INFO: ebs-csi-node-fq5pw started at 2022-11-23 23:18:29 +0000 UTC (0+3 container statuses recorded) Nov 23 23:52:49.181: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:52:49.181: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:52:49.181: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:52:49.181: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-t7n8m started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:49.181: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:49.181: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-d6rv4 started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:49.181: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:49.181: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-lwt6f started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:49.181: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:49.181: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-sxgp9 started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:49.181: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:49.181: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-qxjjp started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:49.181: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:49.181: INFO: cilium-jjmjs started at 2022-11-23 23:18:29 +0000 UTC (1+1 container statuses recorded) Nov 23 23:52:49.181: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:52:49.181: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:52:49.181: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-8m7bj started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:49.181: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:49.764: INFO: Latency metrics for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:52:49.765: INFO: Logging node info for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:52:49.920: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-200.ap-northeast-1.compute.internal 8f32a357-4fd8-4fcd-ab21-8b69dbd931d6 47271 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-200.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-55-200.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d4c764fea6a24b23"} io.cilium.network.ipv4-cilium-host:172.20.61.17 io.cilium.network.ipv4-health-ip:172.20.60.108 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:50 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:40:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {e2e.test Update v1 2022-11-23 23:41:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} }]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0d4c764fea6a24b23,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:50 +0000 UTC,LastTransitionTime:2022-11-23 23:18:50 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:50:34 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:50:34 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:50:34 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:50:34 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.200,},NodeAddress{Type:ExternalIP,Address:3.113.30.221,},NodeAddress{Type:Hostname,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-113-30-221.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ef1c5e899b01a14fef22e61702fe5,SystemUUID:ec2ef1c5-e899-b01a-14fe-f22e61702fe5,BootID:021f6d58-0800-4a46-b65e-3513c48dff08,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:52:49.920: INFO: Logging kubelet events for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:52:50.077: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:52:50.239: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-nmvkk started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:50.239: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-xm57c started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:50.239: INFO: coredns-autoscaler-6658b4bf85-gdgxf started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container autoscaler ready: true, restart count 0 Nov 23 23:52:50.239: INFO: coredns-5fcc7b6498-ll578 started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container coredns ready: true, restart count 0 Nov 23 23:52:50.239: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-d427l started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:50.239: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-5477t started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:50.239: INFO: csi-mockplugin-0 started at 2022-11-23 23:50:17 +0000 UTC (0+3 container statuses recorded) Nov 23 23:52:50.239: INFO: Container csi-provisioner ready: false, restart count 0 Nov 23 23:52:50.239: INFO: Container driver-registrar ready: false, restart count 0 Nov 23 23:52:50.239: INFO: Container mock ready: false, restart count 0 Nov 23 23:52:50.239: INFO: test-deployment-56c98d85f9-bwv4n started at 2022-11-23 23:47:56 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container test-deployment ready: true, restart count 0 Nov 23 23:52:50.239: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-5dv4g started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:50.239: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-mwx9j started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:50.239: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-sgmbg started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:50.239: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:50:17 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container csi-attacher ready: false, restart count 0 Nov 23 23:52:50.239: INFO: csi-mockplugin-resizer-0 started at 2022-11-23 23:50:18 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container csi-resizer ready: false, restart count 0 Nov 23 23:52:50.239: INFO: cilium-2lrb9 started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:52:50.239: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:52:50.239: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-pzhpm started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:50.239: INFO: agnhost-primary-csf28 started at 2022-11-23 23:48:05 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container agnhost-primary ready: false, restart count 0 Nov 23 23:52:50.239: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-7rx5c started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: false, restart count 0 Nov 23 23:52:50.239: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-qqhfs started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:50.239: INFO: ebs-csi-node-dl575 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:52:50.239: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:52:50.239: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:52:50.239: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:52:50.239: INFO: test-deployment-d4dfddfbf-pgn86 started at 2022-11-23 23:47:59 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:50.239: INFO: Container test-deployment ready: false, restart count 0 Nov 23 23:52:50.796: INFO: Latency metrics for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:52:50.796: INFO: Logging node info for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:52:50.951: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-63-80.ap-northeast-1.compute.internal 1d4cdbf2-ca1f-431f-b562-a0bd0f489514 48006 0 2022-11-23 23:18:44 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubelet_cleanup:true kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-63-80.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-63-80.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-volume-expand-5707":"ip-172-20-63-80.ap-northeast-1.compute.internal","ebs.csi.aws.com":"i-081735fbd2bce39f2"} io.cilium.network.ipv4-cilium-host:172.20.50.140 io.cilium.network.ipv4-health-ip:172.20.42.94 io.cilium.network.ipv4-pod-cidr:100.96.4.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.4.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:19:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:19:12 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {e2e.test Update v1 2022-11-23 23:41:31 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubelet_cleanup":{}}}} } {kubelet Update v1 2022-11-23 23:49:35 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {kube-controller-manager Update v1 2022-11-23 23:52:49 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.4.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-081735fbd2bce39f2,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.4.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:19:12 +0000 UTC,LastTransitionTime:2022-11-23 23:19:12 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:52:45 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:52:45 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:52:45 +0000 UTC,LastTransitionTime:2022-11-23 23:18:44 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:52:45 +0000 UTC,LastTransitionTime:2022-11-23 23:19:04 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.63.80,},NodeAddress{Type:ExternalIP,Address:13.112.241.172,},NodeAddress{Type:Hostname,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-63-80.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-112-241-172.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29233394293c3c1bf3f4359c382b54,SystemUUID:ec292333-9429-3c3c-1bf3-f4359c382b54,BootID:05304de1-80b3-439c-9eeb-74d1d733f407,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 k8s.gcr.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[docker.io/library/nginx@sha256:e209ac2f37c70c1e0e9873a5f7231e91dcd83fdf1178d8ed36c2ec09974210ba docker.io/library/nginx:latest],SizeBytes:56833911,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/sample-apiserver@sha256:e7fddbaac4c3451da2365ab90bad149d32f11409738034e41e0f460927f7c276 k8s.gcr.io/e2e-test-images/sample-apiserver:1.17.4],SizeBytes:24757245,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf k8s.gcr.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-expand-5707^eecbb243-6b89-11ed-8582-9efe3bc6d457,DevicePath:,},},Config:nil,},} Nov 23 23:52:50.952: INFO: Logging kubelet events for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:52:51.109: INFO: Logging pods the kubelet thinks is on node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:52:51.269: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-v75b5 started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:51.269: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:51.269: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-cfffd started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:51.269: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:51.269: INFO: csi-hostpathplugin-0 started at 2022-11-23 23:51:17 +0000 UTC (0+7 container statuses recorded) Nov 23 23:52:51.269: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:52:51.269: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:52:51.269: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:52:51.269: INFO: Container csi-snapshotter ready: true, restart count 0 Nov 23 23:52:51.269: INFO: Container hostpath ready: true, restart count 0 Nov 23 23:52:51.269: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:52:51.269: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:52:51.269: INFO: cilium-dgm5r started at 2022-11-23 23:18:44 +0000 UTC (1+1 container statuses recorded) Nov 23 23:52:51.269: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:52:51.269: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:52:51.269: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-bjfq9 started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:51.269: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:51.269: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-m5w97 started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:51.269: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:51.269: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-htxtk started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:51.269: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:51.269: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-gfclm started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:51.269: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:51.269: INFO: pod-2402f2ab-3e95-46a9-8611-520ff22c02f6 started at 2022-11-23 23:52:48 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:51.269: INFO: Container write-pod ready: false, restart count 0 Nov 23 23:52:51.269: INFO: server-envvars-94807307-6c8f-428b-b526-5ac0ff5d8765 started at 2022-11-23 23:47:01 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:51.269: INFO: Container srv ready: true, restart count 0 Nov 23 23:52:51.269: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-v7ktq started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:51.269: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:51.269: INFO: ebs-csi-node-hlgh2 started at 2022-11-23 23:18:44 +0000 UTC (0+3 container statuses recorded) Nov 23 23:52:51.269: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:52:51.269: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:52:51.269: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:52:51.269: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-kj97q started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:51.269: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:51.269: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-2k8qq started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:51.269: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:51.269: INFO: cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630-kjtlp started at 2022-11-23 23:41:32 +0000 UTC (0+1 container statuses recorded) Nov 23 23:52:51.269: INFO: Container cleanup40-8e1b6fec-2867-44d8-9e3d-f2da48075630 ready: true, restart count 0 Nov 23 23:52:51.838: INFO: Latency metrics for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:52:51.838: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready �[1mSTEP�[0m: Destroying namespace "fsgroupchangepolicy-7896" for this suite.
Find pod-feb6db0b-4eb5-47f0-bdb8-f95072e56e8c mentions in log files
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\sprovisioning\sshould\sprovision\sstorage\swith\smount\soptions$'
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:180 Nov 23 23:27:12.158: Unexpected error: <*errors.errorString | 0xc002d81660>: { s: "pod \"pod-3a87dff5-7636-455d-ad98-e1f3bdce533c\" is not Running: timed out waiting for the condition", } pod "pod-3a87dff5-7636-455d-ad98-e1f3bdce533c" is not Running: timed out waiting for the condition occurred /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:418
[BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185 �[1mSTEP�[0m: Creating a kubernetes client Nov 23 23:22:08.481: INFO: >>> kubeConfig: /root/.kube/config �[1mSTEP�[0m: Building a namespace api object, basename provisioning W1123 23:22:10.427069 7181 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ Nov 23 23:22:10.427: INFO: No PodSecurityPolicies found; assuming PodSecurityPolicy is disabled. �[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace [It] should provision storage with mount options /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:180 Nov 23 23:22:10.736: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Nov 23 23:22:10.736: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Nov 23 23:22:10.736: INFO: In creating storage class object and pvc objects for driver - sc: &StorageClass{ObjectMeta:{provisioning-8842-e2e-scnfqx2 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Provisioner:ebs.csi.aws.com,Parameters:map[string]string{encrypted: true,type: gp3,},ReclaimPolicy:nil,MountOptions:[],AllowVolumeExpansion:*true,VolumeBindingMode:*WaitForFirstConsumer,AllowedTopologies:[]TopologySelectorTerm{},}, pvc: &PersistentVolumeClaim{ObjectMeta:{ pvc- provisioning-8842 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*provisioning-8842-e2e-scnfqx2,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},}, src-pvc: &PersistentVolumeClaim{ObjectMeta:{ pvc- provisioning-8842 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*provisioning-8842-e2e-scnfqx2,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} �[1mSTEP�[0m: Creating a StorageClass �[1mSTEP�[0m: creating claim=&PersistentVolumeClaim{ObjectMeta:{ pvc- provisioning-8842 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*provisioning-8842-e2e-scnfqx2,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} �[1mSTEP�[0m: creating a pod referring to the class=&StorageClass{ObjectMeta:{provisioning-8842-e2e-scnfqx2 0356bfcc-3073-4163-95c8-d104f1c3a108 1856 0 2022-11-23 23:22:10 +0000 UTC <nil> <nil> map[] map[] [] [] [{e2e.test Update storage.k8s.io/v1 2022-11-23 23:22:10 +0000 UTC FieldsV1 {"f:allowVolumeExpansion":{},"f:mountOptions":{},"f:parameters":{".":{},"f:encrypted":{},"f:type":{}},"f:provisioner":{},"f:reclaimPolicy":{},"f:volumeBindingMode":{}} }]},Provisioner:ebs.csi.aws.com,Parameters:map[string]string{encrypted: true,type: gp3,},ReclaimPolicy:*Delete,MountOptions:[dirsync],AllowVolumeExpansion:*true,VolumeBindingMode:*WaitForFirstConsumer,AllowedTopologies:[]TopologySelectorTerm{},} claim=&PersistentVolumeClaim{ObjectMeta:{pvc-wvv67 pvc- provisioning-8842 26c8413c-ec32-498b-80cc-bd35e6905ad4 1871 0 2022-11-23 23:22:11 +0000 UTC <nil> <nil> map[] map[] [] [kubernetes.io/pvc-protection] [{e2e.test Update v1 2022-11-23 23:22:11 +0000 UTC FieldsV1 {"f:metadata":{"f:generateName":{}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:storageClassName":{},"f:volumeMode":{}}} }]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*provisioning-8842-e2e-scnfqx2,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,},Status:PersistentVolumeClaimStatus{Phase:Pending,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} Nov 23 23:27:12.157: FAIL: Unexpected error: <*errors.errorString | 0xc002d81660>: { s: "pod \"pod-3a87dff5-7636-455d-ad98-e1f3bdce533c\" is not Running: timed out waiting for the condition", } pod "pod-3a87dff5-7636-455d-ad98-e1f3bdce533c" is not Running: timed out waiting for the condition occurred Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.StorageClassTest.TestDynamicProvisioning(0x7973fe8, 0xc0018ed600, 0xc003dc7dc0, 0xc000312700, 0xc0003128c0, 0xc0007f8300, 0x0, 0x0, 0x0, 0x0, ...) /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:418 +0xe7d k8s.io/kubernetes/test/e2e/storage/testsuites.(*provisioningTestSuite).DefineTests.func3() /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:198 +0x245 k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000b16000) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:130 +0x36c k8s.io/kubernetes/test/e2e.TestE2E(0xc000b16000) _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:136 +0x2b testing.tRunner(0xc000b16000, 0x729c8b8) /usr/local/go/src/testing/testing.go:1203 +0xe5 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1248 +0x2b3 Nov 23 23:27:12.158: INFO: deleting claim "provisioning-8842"/"pvc-wvv67" Nov 23 23:27:12.313: INFO: deleting storage class provisioning-8842-e2e-scnfqx2 [AfterEach] [Testpattern: Dynamic PV (default fs)] provisioning /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186 �[1mSTEP�[0m: Collecting events from namespace "provisioning-8842". �[1mSTEP�[0m: Found 5 events. Nov 23 23:27:12.621: INFO: At 2022-11-23 23:22:11 +0000 UTC - event for pvc-wvv67: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Nov 23 23:27:12.621: INFO: At 2022-11-23 23:22:11 +0000 UTC - event for pvc-wvv67: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Nov 23 23:27:12.621: INFO: At 2022-11-23 23:22:11 +0000 UTC - event for pvc-wvv67: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } Provisioning: External provisioner is provisioning volume for claim "provisioning-8842/pvc-wvv67" Nov 23 23:27:12.621: INFO: At 2022-11-23 23:22:21 +0000 UTC - event for pvc-wvv67: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "provisioning-8842-e2e-scnfqx2": rpc error: code = DeadlineExceeded desc = context deadline exceeded Nov 23 23:27:12.621: INFO: At 2022-11-23 23:24:25 +0000 UTC - event for pvc-wvv67: {ebs.csi.aws.com_ebs-csi-controller-764b64d679-r8ttz_8de28a2d-539d-4c85-8e48-3079c4293ade } ProvisioningFailed: failed to provision volume with StorageClass "provisioning-8842-e2e-scnfqx2": rpc error: code = Internal desc = Could not create volume "pvc-26c8413c-ec32-498b-80cc-bd35e6905ad4": could not create volume in EC2: RequestCanceled: request context canceled caused by: context deadline exceeded Nov 23 23:27:12.775: INFO: POD NODE PHASE GRACE CONDITIONS Nov 23 23:27:12.775: INFO: pod-3a87dff5-7636-455d-ad98-e1f3bdce533c Pending [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2022-11-23 23:27:12 +0000 UTC SchedulerError running PreBind plugin "VolumeBinding": binding volumes: failed to check provisioning pvc: could not find v1.PersistentVolumeClaim "provisioning-8842/pvc-wvv67"}] Nov 23 23:27:12.775: INFO: Nov 23 23:27:13.122: INFO: Logging node info for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:27:13.276: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-139.ap-northeast-1.compute.internal e52821e5-e82f-478b-be4f-6a60ba5816ce 12253 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-139.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-35-139.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-9375":"csi-mock-csi-mock-volumes-9375","ebs.csi.aws.com":"i-0af5c992eacc11f5f"} io.cilium.network.ipv4-cilium-host:172.20.62.73 io.cilium.network.ipv4-health-ip:172.20.63.125 io.cilium.network.ipv4-pod-cidr:100.96.2.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.2.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:45 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:51 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:27:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.2.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0af5c992eacc11f5f,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.2.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:51 +0000 UTC,LastTransitionTime:2022-11-23 23:18:51 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:27:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:27:03 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.139,},NodeAddress{Type:ExternalIP,Address:13.231.193.129,},NodeAddress{Type:Hostname,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-139.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-231-193-129.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec218b5e615d578e61275cbc46e5a775,SystemUUID:ec218b5e-615d-578e-6127-5cbc46e5a775,BootID:15a3520e-c72a-46fe-a3d6-be039aa283b7,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:27:13.277: INFO: Logging kubelet events for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:27:13.433: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:27:13.765: INFO: csi-mockplugin-attacher-0 started at 2022-11-23 23:24:37 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:13.765: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:27:13.765: INFO: test-container-pod started at 2022-11-23 23:27:12 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:13.765: INFO: Container webserver ready: false, restart count 0 Nov 23 23:27:13.765: INFO: test-container-pod started at 2022-11-23 23:27:11 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:13.765: INFO: Container webserver ready: true, restart count 0 Nov 23 23:27:13.765: INFO: cilium-dw7qk started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:13.765: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:27:13.765: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:27:13.765: INFO: netserver-0 started at 2022-11-23 23:26:50 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:13.765: INFO: Container webserver ready: true, restart count 0 Nov 23 23:27:13.765: INFO: csi-mockplugin-0 started at 2022-11-23 23:24:37 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:13.765: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:27:13.765: INFO: Container driver-registrar ready: true, restart count 0 Nov 23 23:27:13.765: INFO: Container mock ready: true, restart count 0 Nov 23 23:27:13.765: INFO: netserver-0 started at 2022-11-23 23:26:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:13.765: INFO: Container webserver ready: true, restart count 0 Nov 23 23:27:13.765: INFO: ebs-csi-node-vbdc7 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:13.765: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:13.765: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:13.765: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:27:13.765: INFO: nodeport-test-89rgk started at 2022-11-23 23:27:10 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:13.765: INFO: Container nodeport-test ready: true, restart count 0 Nov 23 23:27:14.289: INFO: Latency metrics for node ip-172-20-35-139.ap-northeast-1.compute.internal Nov 23 23:27:14.289: INFO: Logging node info for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:27:14.442: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-35-241.ap-northeast-1.compute.internal 86ed4d71-4560-48af-9e88-d8d85c882f8b 6905 0 2022-11-23 23:16:31 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-35-241.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:master node-role.kubernetes.io/control-plane: node-role.kubernetes.io/master: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-027602f60f12db3f0"} io.cilium.network.ipv4-cilium-host:172.20.42.20 io.cilium.network.ipv4-health-ip:172.20.32.68 io.cilium.network.ipv4-pod-cidr:100.96.0.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-23 23:16:30 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {protokube Update v1 2022-11-23 23:16:55 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kops-controller Update v1 2022-11-23 23:17:04 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/master":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:17:21 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.0.0/24\"":{}},"f:taints":{}}} } {cilium-agent Update v1 2022-11-23 23:18:37 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:18:59 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.0.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-027602f60f12db3f0,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.0.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892301824 0} {<nil>} 3801076Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787444224 0} {<nil>} 3698676Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:42 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:16:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:24:21 +0000 UTC,LastTransitionTime:2022-11-23 23:17:21 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.35.241,},NodeAddress{Type:ExternalIP,Address:54.238.84.104,},NodeAddress{Type:Hostname,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-35-241.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-84-104.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2adfc78fc274df02e3c085caa34395,SystemUUID:ec2adfc7-8fc2-74df-02e3-c085caa34395,BootID:e3997507-167a-4ec7-a6ec-7f97c5d886f5,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager@sha256:a91fdaf9b988943a9c73d422348c2383c08dfc2566d4124a39a1b3d785018720 registry.k8s.io/etcdadm/etcd-manager:v3.0.20220831],SizeBytes:216515768,},ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.16],SizeBytes:129618394,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.16],SizeBytes:123306305,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.16],SizeBytes:53956920,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.26.0-alpha.1],SizeBytes:42904862,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.26.0-alpha.1],SizeBytes:41478108,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[quay.io/cilium/operator@sha256:e2b26376a904190c73498e04bd5c1298a4c5239d50feab024074e4516381890f quay.io/cilium/operator:v1.11.11],SizeBytes:24003684,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.26.0-alpha.1],SizeBytes:4955301,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:27:14.443: INFO: Logging kubelet events for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:27:14.599: INFO: Logging pods the kubelet thinks is on node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:27:14.924: INFO: cilium-operator-587dd66f87-j2km7 started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:14.924: INFO: Container cilium-operator ready: true, restart count 1 Nov 23 23:27:14.924: INFO: ebs-csi-controller-764b64d679-r8ttz started at 2022-11-23 23:17:03 +0000 UTC (0+5 container statuses recorded) Nov 23 23:27:14.924: INFO: Container csi-attacher ready: true, restart count 0 Nov 23 23:27:14.924: INFO: Container csi-provisioner ready: true, restart count 0 Nov 23 23:27:14.924: INFO: Container csi-resizer ready: true, restart count 0 Nov 23 23:27:14.924: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:14.924: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:14.924: INFO: etcd-manager-events-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:14.924: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:27:14.924: INFO: cilium-qvdj2 started at 2022-11-23 23:17:02 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:14.925: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:27:14.925: INFO: Container cilium-agent ready: true, restart count 1 Nov 23 23:27:14.925: INFO: etcd-manager-main-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:14.925: INFO: Container etcd-manager ready: true, restart count 0 Nov 23 23:27:14.925: INFO: kube-apiserver-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+2 container statuses recorded) Nov 23 23:27:14.925: INFO: Container healthcheck ready: true, restart count 0 Nov 23 23:27:14.925: INFO: Container kube-apiserver ready: true, restart count 1 Nov 23 23:27:14.925: INFO: ebs-csi-node-94jwj started at 2022-11-23 23:17:02 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:14.925: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:14.925: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:14.925: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:27:14.925: INFO: kops-controller-fznlr started at 2022-11-23 23:17:02 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:14.925: INFO: Container kops-controller ready: true, restart count 0 Nov 23 23:27:14.925: INFO: dns-controller-845449865f-zwg9n started at 2022-11-23 23:17:03 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:14.925: INFO: Container dns-controller ready: true, restart count 0 Nov 23 23:27:14.925: INFO: kube-controller-manager-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:14.925: INFO: Container kube-controller-manager ready: true, restart count 2 Nov 23 23:27:14.925: INFO: kube-scheduler-ip-172-20-35-241.ap-northeast-1.compute.internal started at 2022-11-23 23:15:47 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:14.925: INFO: Container kube-scheduler ready: true, restart count 0 Nov 23 23:27:15.495: INFO: Latency metrics for node ip-172-20-35-241.ap-northeast-1.compute.internal Nov 23 23:27:15.495: INFO: Logging node info for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:27:15.649: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-42-64.ap-northeast-1.compute.internal 45916c04-689e-47fa-b6fb-bdbad0784fcd 9817 0 2022-11-23 23:18:28 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-42-64.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-42-64.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-049ec10a59a1d20e4"} io.cilium.network.ipv4-cilium-host:172.20.51.104 io.cilium.network.ipv4-health-ip:172.20.34.63 io.cilium.network.ipv4-pod-cidr:100.96.3.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kube-controller-manager Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.3.0/24\"":{}}}} } {kubelet Update v1 2022-11-23 23:18:28 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {cilium-agent Update v1 2022-11-23 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:56 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:22:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.3.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-049ec10a59a1d20e4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:56 +0000 UTC,LastTransitionTime:2022-11-23 23:18:56 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:28 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:25:58 +0000 UTC,LastTransitionTime:2022-11-23 23:18:48 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.42.64,},NodeAddress{Type:ExternalIP,Address:54.238.96.164,},NodeAddress{Type:Hostname,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-42-64.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-238-96-164.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec25c6eb725883a2c2916889bd12be88,SystemUUID:ec25c6eb-7258-83a2-c291-6889bd12be88,BootID:8214bf1e-ee6e-4136-8902-31d30749be64,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonroot@sha256:4051e85640c22f8e00c34dbd273576fc9e1e2829992656588062be9c0f69b04b k8s.gcr.io/e2e-test-images/nonroot:1.1],SizeBytes:17748448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:27:15.649: INFO: Logging kubelet events for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:27:15.810: INFO: Logging pods the kubelet thinks is on node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:27:15.978: INFO: coredns-5fcc7b6498-s4jjd started at 2022-11-23 23:19:06 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:15.978: INFO: Container coredns ready: true, restart count 0 Nov 23 23:27:15.978: INFO: foo-tz8t2 started at 2022-11-23 23:26:57 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:15.978: INFO: Container c ready: true, restart count 0 Nov 23 23:27:15.978: INFO: netserver-1 started at 2022-11-23 23:26:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:15.978: INFO: Container webserver ready: true, restart count 0 Nov 23 23:27:15.978: INFO: pod-subpath-test-preprovisionedpv-2d76 started at 2022-11-23 23:27:03 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:15.978: INFO: Init container init-volume-preprovisionedpv-2d76 ready: true, restart count 0 Nov 23 23:27:15.978: INFO: Container test-container-subpath-preprovisionedpv-2d76 ready: true, restart count 0 Nov 23 23:27:15.978: INFO: hostexec-ip-172-20-42-64.ap-northeast-1.compute.internal-d97wc started at 2022-11-23 23:26:51 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:15.978: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:27:15.978: INFO: ebs-csi-node-fq5pw started at 2022-11-23 23:18:29 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:15.978: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:15.978: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:15.978: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:27:15.978: INFO: cilium-jjmjs started at 2022-11-23 23:18:29 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:15.978: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:27:15.978: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:27:15.978: INFO: hostexec-ip-172-20-42-64.ap-northeast-1.compute.internal-v6wzb started at 2022-11-23 23:26:46 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:15.978: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:27:15.978: INFO: netserver-1 started at 2022-11-23 23:26:51 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:15.978: INFO: Container webserver ready: true, restart count 0 Nov 23 23:27:16.535: INFO: Latency metrics for node ip-172-20-42-64.ap-northeast-1.compute.internal Nov 23 23:27:16.535: INFO: Logging node info for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:27:16.689: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-55-200.ap-northeast-1.compute.internal 8f32a357-4fd8-4fcd-ab21-8b69dbd931d6 7902 0 2022-11-23 23:18:22 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-55-200.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d4c764fea6a24b23"} io.cilium.network.ipv4-cilium-host:172.20.61.17 io.cilium.network.ipv4-health-ip:172.20.60.108 io.cilium.network.ipv4-pod-cidr:100.96.1.0/24 node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2022-11-23 23:18:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2022-11-23 23:18:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}}} } {cilium-agent Update v1 2022-11-23 23:18:44 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:io.cilium.network.ipv4-cilium-host":{},"f:io.cilium.network.ipv4-health-ip":{},"f:io.cilium.network.ipv4-pod-cidr":{}}}} status} {cilium-operator Update v1 2022-11-23 23:18:50 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2022-11-23 23:24:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-1a/i-0d4c764fea6a24b23,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-23 23:18:50 +0000 UTC,LastTransitionTime:2022-11-23 23:18:50 +0000 UTC,Reason:CiliumIsUp,Message:Cilium is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-23 23:24:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:22 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-23 23:24:53 +0000 UTC,LastTransitionTime:2022-11-23 23:18:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.55.200,},NodeAddress{Type:ExternalIP,Address:3.113.30.221,},NodeAddress{Type:Hostname,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-55-200.ap-northeast-1.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-113-30-221.ap-northeast-1.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ef1c5e899b01a14fef22e61702fe5,SystemUUID:ec2ef1c5-e899-b01a-14fe-f22e61702fe5,BootID:021f6d58-0800-4a46-b65e-3513c48dff08,KernelVersion:5.15.0-1023-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.4.13,KubeletVersion:v1.22.16,KubeProxyVersion:v1.22.16,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[quay.io/cilium/cilium@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1 quay.io/cilium/cilium:v1.11.11],SizeBytes:160382894,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.16],SizeBytes:105446599,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:7e8bdd271312fd25fc5ff5a8f04727be84044eb3d7d8d03611972a6752e2e11e k8s.gcr.io/e2e-test-images/agnhost:2.39],SizeBytes:51105200,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:21c6eedc2a80bfec5dee669d30d9b5ca83e68cd097f0e210917ae91312e590ff registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.12.0],SizeBytes:28956694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:aa60f453d64dfb3c3fd9e7306f988c36c6352c4f2d956aa90467f2808091effa registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.5],SizeBytes:15209420,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:8e352a029d304ca7431c6507b56800636c321cb52289686a581ab70aaa8a2e2a registry.k8s.io/coredns/coredns:v1.9.3],SizeBytes:14837849,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Nov 23 23:27:16.689: INFO: Logging kubelet events for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:27:16.845: INFO: Logging pods the kubelet thinks is on node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:27:17.030: INFO: ebs-csi-node-dl575 started at 2022-11-23 23:18:23 +0000 UTC (0+3 container statuses recorded) Nov 23 23:27:17.030: INFO: Container ebs-plugin ready: true, restart count 0 Nov 23 23:27:17.030: INFO: Container liveness-probe ready: true, restart count 0 Nov 23 23:27:17.030: INFO: Container node-driver-registrar ready: true, restart count 0 Nov 23 23:27:17.030: INFO: netserver-2 started at 2022-11-23 23:26:49 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:17.030: INFO: Container webserver ready: true, restart count 0 Nov 23 23:27:17.030: INFO: test-pod started at 2022-11-23 23:27:16 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:17.030: INFO: Container nginx ready: false, restart count 0 Nov 23 23:27:17.030: INFO: netserver-2 started at 2022-11-23 23:26:51 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:17.030: INFO: Container webserver ready: true, restart count 0 Nov 23 23:27:17.030: INFO: pod-ephm-test-projected-vkdw started at 2022-11-23 23:25:27 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:17.030: INFO: Container test-container-subpath-projected-vkdw ready: false, restart count 0 Nov 23 23:27:17.030: INFO: hostexec-ip-172-20-55-200.ap-northeast-1.compute.internal-wp6t8 started at 2022-11-23 23:27:07 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:17.030: INFO: Container agnhost-container ready: true, restart count 0 Nov 23 23:27:17.030: INFO: cilium-2lrb9 started at 2022-11-23 23:18:23 +0000 UTC (1+1 container statuses recorded) Nov 23 23:27:17.030: INFO: Init container clean-cilium-state ready: true, restart count 0 Nov 23 23:27:17.030: INFO: Container cilium-agent ready: true, restart count 0 Nov 23 23:27:17.030: INFO: coredns-autoscaler-6658b4bf85-gdgxf started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:17.030: INFO: Container autoscaler ready: true, restart count 0 Nov 23 23:27:17.030: INFO: coredns-5fcc7b6498-ll578 started at 2022-11-23 23:18:42 +0000 UTC (0+1 container statuses recorded) Nov 23 23:27:17.030: INFO: Container coredns ready: true, restart count 0 Nov 23 23:27:17.582: INFO: Latency metrics for node ip-172-20-55-200.ap-northeast-1.compute.internal Nov 23 23:27:17.583: INFO: Logging node info for node ip-172-20-63-80.ap-northeast-1.compute.internal Nov 23 23:27:17.745: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-63-80.ap-northeast-1.compute.internal 1d4cdbf2-ca1f-431f-b562-a0bd0f489514 12148 0 2022-11-23 23:18:44 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-1 failure-domain.beta.kubernetes.io/zone:ap-northeast-1a kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-63-80.ap-northeast-1.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-1a topology.hostpath.csi/node:ip-172-20-63-80.ap-northeast-1.compute.internal topology.kubernetes.io/region:ap-northeast-1 topology.kubernetes.io/zone:ap-northeast-1a] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-081735fbd2bce39f2