Recent runs || View in Spyglass
Result | FAILURE |
Tests | 270 failed / 649 succeeded |
Started | |
Elapsed | 1h7m |
Revision | master |
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sexisting\sdirectories\swhen\sreadOnly\sspecified\sin\sthe\svolumeSource$'
[FAILED] failed to get logs from pod-subpath-test-dynamicpv-4gd5 for test-container-subpath-dynamicpv-4gd5: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1598/pods/pod-subpath-test-dynamicpv-4gd5/log?container=test-container-subpath-dynamicpv-4gd5&previous=false": stream error: stream ID 307; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:47:20.283from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:47:02.188 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:47:02.188 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/27/23 23:47:02.188 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/27/23 23:47:02.188 Mar 27 23:47:02.188: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:250 @ 03/27/23 23:47:02.189 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/27/23 23:47:02.719 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/27/23 23:47:03.069 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/27/23 23:47:03.421 (1.233s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:47:03.421 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:47:03.422 (0s) > Enter [It] should support existing directories when readOnly specified in the volumeSource - test/e2e/storage/testsuites/subpath.go:397 @ 03/27/23 23:47:03.422 Mar 27 23:47:03.422: INFO: Creating resource for dynamic PV Mar 27 23:47:03.422: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass provisioning-1598-e2e-sclktzz - test/e2e/storage/framework/volume_resource.go:102 @ 03/27/23 23:47:03.422 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/27/23 23:47:03.604 Mar 27 23:47:03.604: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod pod-subpath-test-dynamicpv-4gd5 - test/e2e/storage/testsuites/subpath.go:691 @ 03/27/23 23:47:03.962 STEP: Creating a pod to test subpath - test/e2e/framework/pod/output/output.go:243 @ 03/27/23 23:47:03.963 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/27/23 23:47:19.573 Mar 27 23:47:19.749: INFO: Trying to get logs from node i-022003b3a45876d5d pod pod-subpath-test-dynamicpv-4gd5 container test-container-subpath-dynamicpv-4gd5: <nil> Mar 27 23:47:19.924: INFO: Failed to get logs from node "i-022003b3a45876d5d" pod "pod-subpath-test-dynamicpv-4gd5" container "test-container-subpath-dynamicpv-4gd5". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1598/pods/pod-subpath-test-dynamicpv-4gd5/log?container=test-container-subpath-dynamicpv-4gd5&previous=false": stream error: stream ID 307; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/27/23 23:47:19.924 Mar 27 23:47:20.283: INFO: Unexpected error: <*fmt.wrapError | 0xc0035ecd60>: failed to get logs from pod-subpath-test-dynamicpv-4gd5 for test-container-subpath-dynamicpv-4gd5: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1598/pods/pod-subpath-test-dynamicpv-4gd5/log?container=test-container-subpath-dynamicpv-4gd5&previous=false": stream error: stream ID 307; INTERNAL_ERROR; received from peer { msg: "failed to get logs from pod-subpath-test-dynamicpv-4gd5 for test-container-subpath-dynamicpv-4gd5: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1598/pods/pod-subpath-test-dynamicpv-4gd5/log?container=test-container-subpath-dynamicpv-4gd5&previous=false\": stream error: stream ID 307; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc0052c0750>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1598/pods/pod-subpath-test-dynamicpv-4gd5/log?container=test-container-subpath-dynamicpv-4gd5&previous=false", Err: <http2.StreamError>{ StreamID: 307, Code: 2, Cause: <*errors.errorString | 0xc00022b690>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from pod-subpath-test-dynamicpv-4gd5 for test-container-subpath-dynamicpv-4gd5: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1598/pods/pod-subpath-test-dynamicpv-4gd5/log?container=test-container-subpath-dynamicpv-4gd5&previous=false": stream error: stream ID 307; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:47:20.283 < Exit [It] should support existing directories when readOnly specified in the volumeSource - test/e2e/storage/testsuites/subpath.go:397 @ 03/27/23 23:47:20.283 (16.861s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:47:20.283 Mar 27 23:47:20.283: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:47:20.46 (177ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:399 @ 03/27/23 23:47:20.46 STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 03/27/23 23:47:20.46 Mar 27 23:47:20.460: INFO: Deleting pod "pod-subpath-test-dynamicpv-4gd5" in namespace "provisioning-1598" STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/27/23 23:47:20.811 Mar 27 23:47:21.162: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.combjdhn" Mar 27 23:47:21.341: INFO: Waiting up to 5m0s for PersistentVolume pvc-561197b1-00f0-4ef7-b165-ef97d693af5b to get deleted Mar 27 23:47:21.521: INFO: PersistentVolume pvc-561197b1-00f0-4ef7-b165-ef97d693af5b found and phase=Released (179.193147ms) Mar 27 23:47:26.697: INFO: PersistentVolume pvc-561197b1-00f0-4ef7-b165-ef97d693af5b found and phase=Released (5.355098272s) Mar 27 23:47:31.873: INFO: PersistentVolume pvc-561197b1-00f0-4ef7-b165-ef97d693af5b found and phase=Released (10.53128356s) Mar 27 23:47:37.050: INFO: PersistentVolume pvc-561197b1-00f0-4ef7-b165-ef97d693af5b was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/27/23 23:47:37.05 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:399 @ 03/27/23 23:47:37.241 (16.781s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:47:37.241 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:47:37.241 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/27/23 23:47:37.241 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:47:37.241 STEP: Collecting events from namespace "provisioning-1598". - test/e2e/framework/debug/dump.go:42 @ 03/27/23 23:47:37.241 STEP: Found 15 events. - test/e2e/framework/debug/dump.go:46 @ 03/27/23 23:47:37.418 Mar 27 23:47:37.418: INFO: At 2023-03-27 23:47:03 +0000 UTC - event for ebs.csi.aws.combjdhn: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 27 23:47:37.418: INFO: At 2023-03-27 23:47:04 +0000 UTC - event for ebs.csi.aws.combjdhn: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 27 23:47:37.418: INFO: At 2023-03-27 23:47:04 +0000 UTC - event for ebs.csi.aws.combjdhn: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } Provisioning: External provisioner is provisioning volume for claim "provisioning-1598/ebs.csi.aws.combjdhn" Mar 27 23:47:37.418: INFO: At 2023-03-27 23:47:07 +0000 UTC - event for ebs.csi.aws.combjdhn: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } ProvisioningSucceeded: Successfully provisioned volume pvc-561197b1-00f0-4ef7-b165-ef97d693af5b Mar 27 23:47:37.418: INFO: At 2023-03-27 23:47:08 +0000 UTC - event for pod-subpath-test-dynamicpv-4gd5: {default-scheduler } Scheduled: Successfully assigned provisioning-1598/pod-subpath-test-dynamicpv-4gd5 to i-022003b3a45876d5d Mar 27 23:47:37.418: INFO: At 2023-03-27 23:47:10 +0000 UTC - event for pod-subpath-test-dynamicpv-4gd5: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-561197b1-00f0-4ef7-b165-ef97d693af5b" Mar 27 23:47:37.418: INFO: At 2023-03-27 23:47:13 +0000 UTC - event for pod-subpath-test-dynamicpv-4gd5: {kubelet i-022003b3a45876d5d} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 27 23:47:37.418: INFO: At 2023-03-27 23:47:13 +0000 UTC - event for pod-subpath-test-dynamicpv-4gd5: {kubelet i-022003b3a45876d5d} Created: Created container init-volume-dynamicpv-4gd5 Mar 27 23:47:37.418: INFO: At 2023-03-27 23:47:13 +0000 UTC - event for pod-subpath-test-dynamicpv-4gd5: {kubelet i-022003b3a45876d5d} Started: Started container init-volume-dynamicpv-4gd5 Mar 27 23:47:37.418: INFO: At 2023-03-27 23:47:14 +0000 UTC - event for pod-subpath-test-dynamicpv-4gd5: {kubelet i-022003b3a45876d5d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 27 23:47:37.418: INFO: At 2023-03-27 23:47:14 +0000 UTC - event for pod-subpath-test-dynamicpv-4gd5: {kubelet i-022003b3a45876d5d} Created: Created container test-init-subpath-dynamicpv-4gd5 Mar 27 23:47:37.418: INFO: At 2023-03-27 23:47:14 +0000 UTC - event for pod-subpath-test-dynamicpv-4gd5: {kubelet i-022003b3a45876d5d} Started: Started container test-init-subpath-dynamicpv-4gd5 Mar 27 23:47:37.418: INFO: At 2023-03-27 23:47:15 +0000 UTC - event for pod-subpath-test-dynamicpv-4gd5: {kubelet i-022003b3a45876d5d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 27 23:47:37.418: INFO: At 2023-03-27 23:47:15 +0000 UTC - event for pod-subpath-test-dynamicpv-4gd5: {kubelet i-022003b3a45876d5d} Created: Created container test-container-subpath-dynamicpv-4gd5 Mar 27 23:47:37.418: INFO: At 2023-03-27 23:47:15 +0000 UTC - event for pod-subpath-test-dynamicpv-4gd5: {kubelet i-022003b3a45876d5d} Started: Started container test-container-subpath-dynamicpv-4gd5 Mar 27 23:47:37.593: INFO: POD NODE PHASE GRACE CONDITIONS Mar 27 23:47:37.593: INFO: Mar 27 23:47:37.790: INFO: Logging node info for node i-022003b3a45876d5d Mar 27 23:47:37.976: INFO: Node Info: &Node{ObjectMeta:{i-022003b3a45876d5d 20348b58-b610-41a0-accb-ca195a540336 34985 0 2023-03-27 23:33:26 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-022003b3a45876d5d kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-022003b3a45876d5d topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-022003b3a45876d5d"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.49.118/19 projectcalico.org/IPv4IPIPTunnelAddr:100.115.192.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:33:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:33:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:47:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-022003b3a45876d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:33:51 +0000 UTC,LastTransitionTime:2023-03-27 23:33:51 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:47:22 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:47:22 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:47:22 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:47:22 +0000 UTC,LastTransitionTime:2023-03-27 23:33:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.118,},NodeAddress{Type:ExternalIP,Address:54.180.103.89,},NodeAddress{Type:InternalDNS,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-103-89.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccb51411bbbb92946da93b9a452fe,SystemUUID:ec2ccb51-411b-bbb9-2946-da93b9a452fe,BootID:841d91a9-bfa5-4e0f-9279-7c83efeef8ca,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:47:37.976: INFO: Logging kubelet events for node i-022003b3a45876d5d Mar 27 23:47:38.163: INFO: Logging pods the kubelet thinks is on node i-022003b3a45876d5d Mar 27 23:47:38.353: INFO: calico-node-2wjdb started at 2023-03-27 23:33:27 +0000 UTC (3+1 container statuses recorded) Mar 27 23:47:38.353: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:47:38.353: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:47:38.353: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:47:38.353: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:47:38.353: INFO: pod-secrets-f9689818-53f9-4e56-b6f9-b3f9fe8d517b started at 2023-03-27 23:43:23 +0000 UTC (0+3 container statuses recorded) Mar 27 23:47:38.353: INFO: Container creates-volume-test ready: false, restart count 0 Mar 27 23:47:38.353: INFO: Container dels-volume-test ready: false, restart count 0 Mar 27 23:47:38.353: INFO: Container upds-volume-test ready: false, restart count 0 Mar 27 23:47:38.353: INFO: pod-adoption-release started at 2023-03-27 23:47:36 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:38.353: INFO: Container pod-adoption-release ready: false, restart count 0 Mar 27 23:47:38.353: INFO: kube-proxy-i-022003b3a45876d5d started at 2023-03-27 23:33:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:38.353: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:47:38.353: INFO: ebs-csi-node-2knd4 started at 2023-03-27 23:33:27 +0000 UTC (0+3 container statuses recorded) Mar 27 23:47:38.353: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:47:38.353: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:47:38.353: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:47:38.353: INFO: pfpod started at 2023-03-27 23:46:53 +0000 UTC (0+2 container statuses recorded) Mar 27 23:47:38.353: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:47:38.353: INFO: Container readiness ready: false, restart count 0 Mar 27 23:47:38.353: INFO: oidc-discovery-validator started at 2023-03-27 23:47:33 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:38.353: INFO: Container oidc-discovery-validator ready: false, restart count 0 Mar 27 23:47:38.353: INFO: test-rs-c9vn4 started at 2023-03-27 23:47:38 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:38.353: INFO: Container httpd ready: false, restart count 0 Mar 27 23:47:38.353: INFO: externalip-test-59b4m started at 2023-03-27 23:47:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:38.353: INFO: Container externalip-test ready: false, restart count 0 Mar 27 23:47:38.353: INFO: wrong-pod-os started at 2023-03-27 23:47:36 +0000 UTC (0+0 container statuses recorded) Mar 27 23:47:38.353: INFO: dns-test-6b1ef813-47ba-4d05-aaa0-4f1021b2e768 started at 2023-03-27 23:47:36 +0000 UTC (0+3 container statuses recorded) Mar 27 23:47:38.353: INFO: Container jessie-querier ready: false, restart count 0 Mar 27 23:47:38.353: INFO: Container querier ready: false, restart count 0 Mar 27 23:47:38.353: INFO: Container webserver ready: false, restart count 0 Mar 27 23:47:38.353: INFO: test-pod started at 2023-03-27 23:47:35 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:38.353: INFO: Container nginx ready: true, restart count 0 Mar 27 23:47:39.268: INFO: Latency metrics for node i-022003b3a45876d5d Mar 27 23:47:39.268: INFO: Logging node info for node i-05ec0d9f3272223ec Mar 27 23:47:39.444: INFO: Node Info: &Node{ObjectMeta:{i-05ec0d9f3272223ec 365deda4-148e-4593-abdf-3103630fef28 34975 0 2023-03-27 23:25:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-05ec0d9f3272223ec kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-05ec0d9f3272223ec topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-05ec0d9f3272223ec"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.52.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.113.211.128 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:25:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:25:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:47:05 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:47:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-05ec0d9f3272223ec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:25:42 +0000 UTC,LastTransitionTime:2023-03-27 23:25:42 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:47:05 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:47:05 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:47:05 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:47:05 +0000 UTC,LastTransitionTime:2023-03-27 23:25:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.36,},NodeAddress{Type:ExternalIP,Address:3.34.125.198,},NodeAddress{Type:InternalDNS,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-34-125-198.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dea8f8b5e44e8fd6ef34a7ee9507f,SystemUUID:ec2dea8f-8b5e-44e8-fd6e-f34a7ee9507f,BootID:d41641dd-09ee-4029-b1ae-fb0f0a4ef62f,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d,DevicePath:,},},Config:nil,},} Mar 27 23:47:39.445: INFO: Logging kubelet events for node i-05ec0d9f3272223ec Mar 27 23:47:39.661: INFO: Logging pods the kubelet thinks is on node i-05ec0d9f3272223ec Mar 27 23:47:39.848: INFO: test-webserver-855e3a20-bdd3-4ab4-bbaa-d646ee893338 started at 2023-03-27 23:47:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:39.848: INFO: Container test-webserver ready: true, restart count 0 Mar 27 23:47:39.848: INFO: kube-proxy-i-05ec0d9f3272223ec started at 2023-03-27 23:25:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:39.848: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:47:39.848: INFO: hostexec-i-05ec0d9f3272223ec-cmqmm started at 2023-03-27 23:47:26 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:39.848: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:47:39.848: INFO: ebs-csi-node-gjw52 started at 2023-03-27 23:25:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:47:39.848: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:47:39.848: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:47:39.848: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:47:39.848: INFO: coredns-559769c974-m6hh6 started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:39.848: INFO: Container coredns ready: true, restart count 0 Mar 27 23:47:39.848: INFO: inline-volume-tester-kwwd2 started at 2023-03-27 23:43:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:39.848: INFO: Container csi-volume-tester ready: false, restart count 0 Mar 27 23:47:39.848: INFO: csi-mockplugin-0 started at 2023-03-27 23:46:17 +0000 UTC (0+4 container statuses recorded) Mar 27 23:47:39.848: INFO: Container busybox ready: true, restart count 0 Mar 27 23:47:39.848: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:47:39.848: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:47:39.848: INFO: Container mock ready: true, restart count 0 Mar 27 23:47:39.848: INFO: test-webserver-d61a8cc2-59e4-4516-aa86-d6b94f858278 started at 2023-03-27 23:47:23 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:39.848: INFO: Container test-webserver ready: true, restart count 0 Mar 27 23:47:39.848: INFO: calico-node-5lq5m started at 2023-03-27 23:25:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:47:39.848: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:47:39.848: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:47:39.848: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:47:39.848: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:47:39.848: INFO: externalip-test-vs797 started at 2023-03-27 23:47:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:39.848: INFO: Container externalip-test ready: true, restart count 0 Mar 27 23:47:39.848: INFO: pod1 started at 2023-03-27 23:47:39 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:39.848: INFO: Container agnhost-container ready: false, restart count 0 Mar 27 23:47:40.474: INFO: Latency metrics for node i-05ec0d9f3272223ec Mar 27 23:47:40.474: INFO: Logging node info for node i-0b266cca2f40af1c1 Mar 27 23:47:40.651: INFO: Node Info: &Node{ObjectMeta:{i-0b266cca2f40af1c1 5f957448-9afc-4966-b9d6-82332b7a159d 35595 0 2023-03-27 23:29:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0b266cca2f40af1c1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0b266cca2f40af1c1 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-workload-1549":"i-0b266cca2f40af1c1","ebs.csi.aws.com":"i-0b266cca2f40af1c1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.116.128.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:30:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:30:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:47:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0b266cca2f40af1c1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:30:13 +0000 UTC,LastTransitionTime:2023-03-27 23:30:13 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:47:39 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:47:39 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:47:39 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:47:39 +0000 UTC,LastTransitionTime:2023-03-27 23:30:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.36,},NodeAddress{Type:ExternalIP,Address:13.209.22.95,},NodeAddress{Type:InternalDNS,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-209-22-95.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec247601b7b315369bfb310559a7772d,SystemUUID:ec247601-b7b3-1536-9bfb-310559a7772d,BootID:487b59c3-9394-4a90-a6ec-b20c4819fe9e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:47:40.651: INFO: Logging kubelet events for node i-0b266cca2f40af1c1 Mar 27 23:47:40.839: INFO: Logging pods the kubelet thinks is on node i-0b266cca2f40af1c1 Mar 27 23:47:41.051: INFO: pod-configmaps-469afe86-b23e-4133-b521-f7d9cffdedcf started at 2023-03-27 23:46:30 +0000 UTC (0+2 container statuses recorded) Mar 27 23:47:41.051: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:47:41.051: INFO: Container configmap-volume-binary-test ready: false, restart count 0 Mar 27 23:47:41.051: INFO: csi-mockplugin-0 started at 2023-03-27 23:47:15 +0000 UTC (0+3 container statuses recorded) Mar 27 23:47:41.051: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:47:41.051: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:47:41.051: INFO: Container mock ready: true, restart count 0 Mar 27 23:47:41.051: INFO: ebs-csi-node-ztjm9 started at 2023-03-27 23:29:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:47:41.051: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:47:41.051: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:47:41.051: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:47:41.051: INFO: pod-configmaps-cb64ad83-3af4-4c45-9bea-66eb9f5eeb11 started at 2023-03-27 23:44:25 +0000 UTC (0+3 container statuses recorded) Mar 27 23:47:41.051: INFO: Container createcm-volume-test ready: true, restart count 0 Mar 27 23:47:41.051: INFO: Container delcm-volume-test ready: true, restart count 0 Mar 27 23:47:41.051: INFO: Container updcm-volume-test ready: true, restart count 0 Mar 27 23:47:41.051: INFO: csi-mockplugin-attacher-0 started at 2023-03-27 23:47:15 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:41.051: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:47:41.051: INFO: kube-proxy-i-0b266cca2f40af1c1 started at 2023-03-27 23:29:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:41.051: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:47:41.051: INFO: calico-node-nmfrn started at 2023-03-27 23:29:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:47:41.051: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:47:41.051: INFO: Init container install-cni ready: true, restart count 1 Mar 27 23:47:41.051: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:47:41.051: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:47:41.732: INFO: Latency metrics for node i-0b266cca2f40af1c1 Mar 27 23:47:41.732: INFO: Logging node info for node i-0d2b1184f62de758a Mar 27 23:47:41.909: INFO: Node Info: &Node{ObjectMeta:{i-0d2b1184f62de758a 2e42ccb1-9580-4126-b5d6-05a5f98403c5 31490 0 2023-03-27 23:17:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d2b1184f62de758a kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d2b1184f62de758a"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.6/19 projectcalico.org/IPv4IPIPTunnelAddr:100.120.150.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-27 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-27 23:17:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:18:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:45:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0d2b1184f62de758a,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:18:49 +0000 UTC,LastTransitionTime:2023-03-27 23:18:49 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:45:46 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:45:46 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:45:46 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:45:46 +0000 UTC,LastTransitionTime:2023-03-27 23:18:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.6,},NodeAddress{Type:ExternalIP,Address:3.35.11.126,},NodeAddress{Type:InternalDNS,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-35-11-126.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2781baf4898d15db3c853bf5b6d100,SystemUUID:ec2781ba-f489-8d15-db3c-853bf5b6d100,BootID:5e905df0-494e-4050-b6d7-400763c3dd66,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45208814,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43629012,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074727,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:47:41.909: INFO: Logging kubelet events for node i-0d2b1184f62de758a Mar 27 23:47:42.099: INFO: Logging pods the kubelet thinks is on node i-0d2b1184f62de758a Mar 27 23:47:42.284: INFO: kube-scheduler-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:42.284: INFO: Container kube-scheduler ready: true, restart count 0 Mar 27 23:47:42.284: INFO: dns-controller-69987775c6-rzzjj started at 2023-03-27 23:18:25 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:42.284: INFO: Container dns-controller ready: true, restart count 0 Mar 27 23:47:42.284: INFO: calico-node-scbv2 started at 2023-03-27 23:18:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:47:42.284: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:47:42.284: INFO: ebs-csi-node-nvqcx started at 2023-03-27 23:18:29 +0000 UTC (0+3 container statuses recorded) Mar 27 23:47:42.284: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:47:42.284: INFO: kops-controller-w4b7t started at 2023-03-27 23:18:29 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:42.284: INFO: Container kops-controller ready: true, restart count 0 Mar 27 23:47:42.284: INFO: calico-kube-controllers-55f488656-vj6gh started at 2023-03-27 23:18:57 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:42.284: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 27 23:47:42.284: INFO: ebs-csi-controller-79f984545f-42vd6 started at 2023-03-27 23:19:48 +0000 UTC (0+5 container statuses recorded) Mar 27 23:47:42.284: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:47:42.284: INFO: etcd-manager-events-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:47:42.284: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:47:42.284: INFO: etcd-manager-main-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:47:42.284: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:47:42.284: INFO: kube-apiserver-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+2 container statuses recorded) Mar 27 23:47:42.284: INFO: Container healthcheck ready: true, restart count 0 Mar 27 23:47:42.284: INFO: Container kube-apiserver ready: true, restart count 3 Mar 27 23:47:42.284: INFO: kube-controller-manager-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:42.284: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 27 23:47:42.284: INFO: kube-proxy-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:42.284: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:47:42.284: INFO: aws-cloud-controller-manager-776ld started at 2023-03-27 23:18:28 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:42.284: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 27 23:47:42.974: INFO: Latency metrics for node i-0d2b1184f62de758a Mar 27 23:47:42.974: INFO: Logging node info for node i-0e1f2cd7cf96d3020 Mar 27 23:47:43.150: INFO: Node Info: &Node{ObjectMeta:{i-0e1f2cd7cf96d3020 efcf1e66-db0e-4256-a849-b9c377eef04e 35658 0 2023-03-27 23:22:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e1f2cd7cf96d3020 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0e1f2cd7cf96d3020 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-volume-expand-5306":"i-0e1f2cd7cf96d3020","csi-mock-csi-mock-volumes-expansion-4700":"i-0e1f2cd7cf96d3020","ebs.csi.aws.com":"i-0e1f2cd7cf96d3020"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.120/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.86.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:22:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:22:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:46:15 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:47:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0e1f2cd7cf96d3020,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:22:52 +0000 UTC,LastTransitionTime:2023-03-27 23:22:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:46:15 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:46:15 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:46:15 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:46:15 +0000 UTC,LastTransitionTime:2023-03-27 23:22:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.120,},NodeAddress{Type:ExternalIP,Address:15.164.224.157,},NodeAddress{Type:InternalDNS,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-164-224-157.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29484c87c97fb3d8a52788d5971660,SystemUUID:ec29484c-87c9-7fb3-d8a5-2788d5971660,BootID:b73c540f-bd49-470a-a7b3-08c31cc8c2fd,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4700^301e3bbe-ccf9-11ed-a882-f233d05cf674],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4700^301e3bbe-ccf9-11ed-a882-f233d05cf674,DevicePath:,},},Config:nil,},} Mar 27 23:47:43.150: INFO: Logging kubelet events for node i-0e1f2cd7cf96d3020 Mar 27 23:47:43.333: INFO: Logging pods the kubelet thinks is on node i-0e1f2cd7cf96d3020 Mar 27 23:47:43.516: INFO: coredns-559769c974-scn7m started at 2023-03-27 23:30:58 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:43.516: INFO: Container coredns ready: true, restart count 0 Mar 27 23:47:43.516: INFO: pvc-volume-tester-cwxsw started at 2023-03-27 23:43:35 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:43.516: INFO: Container volume-tester ready: true, restart count 0 Mar 27 23:47:43.516: INFO: startup-b53aec93-16d6-4687-ad53-d8287455fded started at 2023-03-27 23:47:21 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:43.516: INFO: Container busybox ready: false, restart count 0 Mar 27 23:47:43.516: INFO: ebs-csi-node-vcn4h started at 2023-03-27 23:22:28 +0000 UTC (0+3 container statuses recorded) Mar 27 23:47:43.516: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:47:43.516: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:47:43.516: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:47:43.516: INFO: pod-client started at 2023-03-27 23:47:39 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:43.516: INFO: Container pod-client ready: true, restart count 0 Mar 27 23:47:43.516: INFO: kube-proxy-i-0e1f2cd7cf96d3020 started at 2023-03-27 23:22:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:43.516: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:47:43.516: INFO: calico-node-9fzr7 started at 2023-03-27 23:22:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:47:43.516: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:47:43.516: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:47:43.516: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:47:43.516: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:47:43.516: INFO: coredns-autoscaler-7cb5c5b969-2k8xf started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:43.516: INFO: Container autoscaler ready: true, restart count 0 Mar 27 23:47:43.516: INFO: csi-mockplugin-resizer-0 started at 2023-03-27 23:43:31 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:43.516: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:47:43.516: INFO: csi-mockplugin-attacher-0 started at 2023-03-27 23:43:31 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:43.516: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:47:43.516: INFO: csi-mockplugin-0 started at 2023-03-27 23:43:31 +0000 UTC (0+3 container statuses recorded) Mar 27 23:47:43.516: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:47:43.516: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:47:43.516: INFO: Container mock ready: true, restart count 0 Mar 27 23:47:43.516: INFO: hostexec-i-0e1f2cd7cf96d3020-2hnrx started at 2023-03-27 23:47:31 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:43.516: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:47:43.516: INFO: csi-hostpathplugin-0 started at 2023-03-27 23:47:39 +0000 UTC (0+7 container statuses recorded) Mar 27 23:47:43.516: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:47:43.516: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:47:43.516: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:47:43.516: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 27 23:47:43.516: INFO: Container hostpath ready: true, restart count 0 Mar 27 23:47:43.516: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:47:43.516: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:47:43.516: INFO: pod-ce587519-bca8-43f1-96ef-372ab54eab61 started at 2023-03-27 23:47:42 +0000 UTC (0+1 container statuses recorded) Mar 27 23:47:43.516: INFO: Container write-pod ready: false, restart count 0 Mar 27 23:47:44.203: INFO: Latency metrics for node i-0e1f2cd7cf96d3020 END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:47:44.203 (6.963s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/27/23 23:47:44.204 (6.963s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/27/23 23:47:44.204 STEP: Destroying namespace "provisioning-1598" for this suite. - test/e2e/framework/framework.go:351 @ 03/27/23 23:47:44.204 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/27/23 23:47:44.381 (178ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:47:44.381 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:47:44.381 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sexisting\sdirectory$'
[FAILED] failed to get logs from pod-subpath-test-dynamicpv-9nxg for test-container-volume-dynamicpv-9nxg: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-9732/pods/pod-subpath-test-dynamicpv-9nxg/log?container=test-container-volume-dynamicpv-9nxg&previous=false": stream error: stream ID 2711; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:53:07.687from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:52:49.324 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:52:49.324 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/27/23 23:52:49.324 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/27/23 23:52:49.324 Mar 27 23:52:49.324: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:250 @ 03/27/23 23:52:49.326 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/27/23 23:52:49.893 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/27/23 23:52:50.269 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/27/23 23:52:50.626 (1.302s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:52:50.626 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:52:50.626 (0s) > Enter [It] should support existing directory - test/e2e/storage/testsuites/subpath.go:205 @ 03/27/23 23:52:50.626 Mar 27 23:52:50.626: INFO: Creating resource for dynamic PV Mar 27 23:52:50.626: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass provisioning-9732-e2e-sc47nnh - test/e2e/storage/framework/volume_resource.go:102 @ 03/27/23 23:52:50.626 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/27/23 23:52:50.806 Mar 27 23:52:50.806: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod pod-subpath-test-dynamicpv-9nxg - test/e2e/storage/testsuites/subpath.go:691 @ 03/27/23 23:52:51.17 STEP: Creating a pod to test subpath - test/e2e/framework/pod/output/output.go:243 @ 03/27/23 23:52:51.17 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/27/23 23:53:06.967 Mar 27 23:53:07.144: INFO: Trying to get logs from node i-022003b3a45876d5d pod pod-subpath-test-dynamicpv-9nxg container test-container-volume-dynamicpv-9nxg: <nil> Mar 27 23:53:07.322: INFO: Failed to get logs from node "i-022003b3a45876d5d" pod "pod-subpath-test-dynamicpv-9nxg" container "test-container-volume-dynamicpv-9nxg". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-9732/pods/pod-subpath-test-dynamicpv-9nxg/log?container=test-container-volume-dynamicpv-9nxg&previous=false": stream error: stream ID 2711; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/27/23 23:53:07.323 Mar 27 23:53:07.687: INFO: Unexpected error: <*fmt.wrapError | 0xc00589bc80>: failed to get logs from pod-subpath-test-dynamicpv-9nxg for test-container-volume-dynamicpv-9nxg: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-9732/pods/pod-subpath-test-dynamicpv-9nxg/log?container=test-container-volume-dynamicpv-9nxg&previous=false": stream error: stream ID 2711; INTERNAL_ERROR; received from peer { msg: "failed to get logs from pod-subpath-test-dynamicpv-9nxg for test-container-volume-dynamicpv-9nxg: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-9732/pods/pod-subpath-test-dynamicpv-9nxg/log?container=test-container-volume-dynamicpv-9nxg&previous=false\": stream error: stream ID 2711; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc0057da480>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-9732/pods/pod-subpath-test-dynamicpv-9nxg/log?container=test-container-volume-dynamicpv-9nxg&previous=false", Err: <http2.StreamError>{ StreamID: 2711, Code: 2, Cause: <*errors.errorString | 0xc0001b3650>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from pod-subpath-test-dynamicpv-9nxg for test-container-volume-dynamicpv-9nxg: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-9732/pods/pod-subpath-test-dynamicpv-9nxg/log?container=test-container-volume-dynamicpv-9nxg&previous=false": stream error: stream ID 2711; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:53:07.687 < Exit [It] should support existing directory - test/e2e/storage/testsuites/subpath.go:205 @ 03/27/23 23:53:07.687 (17.062s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:53:07.687 Mar 27 23:53:07.687: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:53:07.865 (178ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:207 @ 03/27/23 23:53:07.865 STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 03/27/23 23:53:07.865 Mar 27 23:53:07.865: INFO: Deleting pod "pod-subpath-test-dynamicpv-9nxg" in namespace "provisioning-9732" STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/27/23 23:53:08.215 Mar 27 23:53:08.567: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comrjls5" Mar 27 23:53:08.746: INFO: Waiting up to 5m0s for PersistentVolume pvc-cbf5491c-9350-4725-b8f7-de0ba9b74548 to get deleted Mar 27 23:53:08.922: INFO: PersistentVolume pvc-cbf5491c-9350-4725-b8f7-de0ba9b74548 found and phase=Released (175.999061ms) Mar 27 23:53:14.099: INFO: PersistentVolume pvc-cbf5491c-9350-4725-b8f7-de0ba9b74548 found and phase=Released (5.353694027s) Mar 27 23:53:19.277: INFO: PersistentVolume pvc-cbf5491c-9350-4725-b8f7-de0ba9b74548 was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/27/23 23:53:19.277 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:207 @ 03/27/23 23:53:19.487 (11.622s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:53:19.487 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:53:19.487 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/27/23 23:53:19.487 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:53:19.487 STEP: Collecting events from namespace "provisioning-9732". - test/e2e/framework/debug/dump.go:42 @ 03/27/23 23:53:19.488 STEP: Found 18 events. - test/e2e/framework/debug/dump.go:46 @ 03/27/23 23:53:19.664 Mar 27 23:53:19.664: INFO: At 2023-03-27 23:52:50 +0000 UTC - event for ebs.csi.aws.comrjls5: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 27 23:53:19.664: INFO: At 2023-03-27 23:52:51 +0000 UTC - event for ebs.csi.aws.comrjls5: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 27 23:53:19.664: INFO: At 2023-03-27 23:52:52 +0000 UTC - event for ebs.csi.aws.comrjls5: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } Provisioning: External provisioner is provisioning volume for claim "provisioning-9732/ebs.csi.aws.comrjls5" Mar 27 23:53:19.664: INFO: At 2023-03-27 23:52:55 +0000 UTC - event for ebs.csi.aws.comrjls5: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } ProvisioningSucceeded: Successfully provisioned volume pvc-cbf5491c-9350-4725-b8f7-de0ba9b74548 Mar 27 23:53:19.664: INFO: At 2023-03-27 23:52:56 +0000 UTC - event for pod-subpath-test-dynamicpv-9nxg: {default-scheduler } Scheduled: Successfully assigned provisioning-9732/pod-subpath-test-dynamicpv-9nxg to i-022003b3a45876d5d Mar 27 23:53:19.664: INFO: At 2023-03-27 23:52:59 +0000 UTC - event for pod-subpath-test-dynamicpv-9nxg: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-cbf5491c-9350-4725-b8f7-de0ba9b74548" Mar 27 23:53:19.664: INFO: At 2023-03-27 23:53:01 +0000 UTC - event for pod-subpath-test-dynamicpv-9nxg: {kubelet i-022003b3a45876d5d} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 27 23:53:19.664: INFO: At 2023-03-27 23:53:01 +0000 UTC - event for pod-subpath-test-dynamicpv-9nxg: {kubelet i-022003b3a45876d5d} Created: Created container init-volume-dynamicpv-9nxg Mar 27 23:53:19.664: INFO: At 2023-03-27 23:53:01 +0000 UTC - event for pod-subpath-test-dynamicpv-9nxg: {kubelet i-022003b3a45876d5d} Started: Started container init-volume-dynamicpv-9nxg Mar 27 23:53:19.664: INFO: At 2023-03-27 23:53:02 +0000 UTC - event for pod-subpath-test-dynamicpv-9nxg: {kubelet i-022003b3a45876d5d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 27 23:53:19.664: INFO: At 2023-03-27 23:53:02 +0000 UTC - event for pod-subpath-test-dynamicpv-9nxg: {kubelet i-022003b3a45876d5d} Created: Created container test-init-subpath-dynamicpv-9nxg Mar 27 23:53:19.664: INFO: At 2023-03-27 23:53:02 +0000 UTC - event for pod-subpath-test-dynamicpv-9nxg: {kubelet i-022003b3a45876d5d} Started: Started container test-init-subpath-dynamicpv-9nxg Mar 27 23:53:19.664: INFO: At 2023-03-27 23:53:03 +0000 UTC - event for pod-subpath-test-dynamicpv-9nxg: {kubelet i-022003b3a45876d5d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 27 23:53:19.664: INFO: At 2023-03-27 23:53:03 +0000 UTC - event for pod-subpath-test-dynamicpv-9nxg: {kubelet i-022003b3a45876d5d} Created: Created container test-container-subpath-dynamicpv-9nxg Mar 27 23:53:19.664: INFO: At 2023-03-27 23:53:03 +0000 UTC - event for pod-subpath-test-dynamicpv-9nxg: {kubelet i-022003b3a45876d5d} Started: Started container test-container-subpath-dynamicpv-9nxg Mar 27 23:53:19.664: INFO: At 2023-03-27 23:53:03 +0000 UTC - event for pod-subpath-test-dynamicpv-9nxg: {kubelet i-022003b3a45876d5d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 27 23:53:19.664: INFO: At 2023-03-27 23:53:03 +0000 UTC - event for pod-subpath-test-dynamicpv-9nxg: {kubelet i-022003b3a45876d5d} Created: Created container test-container-volume-dynamicpv-9nxg Mar 27 23:53:19.664: INFO: At 2023-03-27 23:53:03 +0000 UTC - event for pod-subpath-test-dynamicpv-9nxg: {kubelet i-022003b3a45876d5d} Started: Started container test-container-volume-dynamicpv-9nxg Mar 27 23:53:19.840: INFO: POD NODE PHASE GRACE CONDITIONS Mar 27 23:53:19.840: INFO: Mar 27 23:53:20.018: INFO: Logging node info for node i-022003b3a45876d5d Mar 27 23:53:20.368: INFO: Node Info: &Node{ObjectMeta:{i-022003b3a45876d5d 20348b58-b610-41a0-accb-ca195a540336 53393 0 2023-03-27 23:33:26 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-022003b3a45876d5d kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-022003b3a45876d5d topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-022003b3a45876d5d"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.49.118/19 projectcalico.org/IPv4IPIPTunnelAddr:100.115.192.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:33:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:33:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:53:08 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-27 23:53:10 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-022003b3a45876d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:33:51 +0000 UTC,LastTransitionTime:2023-03-27 23:33:51 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:08 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:08 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:08 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:53:08 +0000 UTC,LastTransitionTime:2023-03-27 23:33:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.118,},NodeAddress{Type:ExternalIP,Address:54.180.103.89,},NodeAddress{Type:InternalDNS,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-103-89.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccb51411bbbb92946da93b9a452fe,SystemUUID:ec2ccb51-411b-bbb9-2946-da93b9a452fe,BootID:841d91a9-bfa5-4e0f-9279-7c83efeef8ca,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-032846faeb20c668e kubernetes.io/csi/ebs.csi.aws.com^vol-0d0441d11d4bdc02e],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-032846faeb20c668e,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0d0441d11d4bdc02e,DevicePath:,},},Config:nil,},} Mar 27 23:53:20.368: INFO: Logging kubelet events for node i-022003b3a45876d5d Mar 27 23:53:20.560: INFO: Logging pods the kubelet thinks is on node i-022003b3a45876d5d Mar 27 23:53:20.742: INFO: calico-node-2wjdb started at 2023-03-27 23:33:27 +0000 UTC (3+1 container statuses recorded) Mar 27 23:53:20.742: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:53:20.742: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:53:20.742: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:53:20.742: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:53:20.742: INFO: webserver-pod started at 2023-03-27 23:52:55 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:20.742: INFO: Container agnhost ready: true, restart count 0 Mar 27 23:53:20.742: INFO: inline-volume-p4k8g started at 2023-03-27 23:52:15 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:20.742: INFO: Container volume-tester ready: false, restart count 0 Mar 27 23:53:20.742: INFO: pod-prestop-hook-362b99a3-2ca4-4164-b33a-0c35131611f0 started at 2023-03-27 23:52:56 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:20.742: INFO: Container nginx ready: true, restart count 0 Mar 27 23:53:20.742: INFO: inline-volume-tester2-vqpkp started at 2023-03-27 23:53:07 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:20.742: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 27 23:53:20.742: INFO: kube-proxy-i-022003b3a45876d5d started at 2023-03-27 23:33:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:20.742: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:53:20.742: INFO: ebs-csi-node-2knd4 started at 2023-03-27 23:33:27 +0000 UTC (0+3 container statuses recorded) Mar 27 23:53:20.742: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:53:20.742: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:53:20.742: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:53:20.742: INFO: pfpod started at 2023-03-27 23:52:53 +0000 UTC (0+2 container statuses recorded) Mar 27 23:53:20.742: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:53:20.742: INFO: Container readiness ready: false, restart count 0 Mar 27 23:53:20.742: INFO: pfpod started at 2023-03-27 23:53:13 +0000 UTC (0+2 container statuses recorded) Mar 27 23:53:20.742: INFO: Container portforwardtester ready: true, restart count 0 Mar 27 23:53:20.742: INFO: Container readiness ready: true, restart count 0 Mar 27 23:53:20.742: INFO: pod-3c7b65ab-744f-4a60-b98d-d02d93a90be0 started at 2023-03-27 23:52:51 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:20.742: INFO: Container write-pod ready: true, restart count 0 Mar 27 23:53:21.367: INFO: Latency metrics for node i-022003b3a45876d5d Mar 27 23:53:21.367: INFO: Logging node info for node i-05ec0d9f3272223ec Mar 27 23:53:21.543: INFO: Node Info: &Node{ObjectMeta:{i-05ec0d9f3272223ec 365deda4-148e-4593-abdf-3103630fef28 53528 0 2023-03-27 23:25:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-05ec0d9f3272223ec kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-05ec0d9f3272223ec topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-volume-6142":"i-05ec0d9f3272223ec","ebs.csi.aws.com":"i-05ec0d9f3272223ec"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.52.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.113.211.128 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:25:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:25:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:53:02 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:53:15 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-05ec0d9f3272223ec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:25:42 +0000 UTC,LastTransitionTime:2023-03-27 23:25:42 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:02 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:02 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:02 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:53:02 +0000 UTC,LastTransitionTime:2023-03-27 23:25:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.36,},NodeAddress{Type:ExternalIP,Address:3.34.125.198,},NodeAddress{Type:InternalDNS,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-34-125-198.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dea8f8b5e44e8fd6ef34a7ee9507f,SystemUUID:ec2dea8f-8b5e-44e8-fd6e-f34a7ee9507f,BootID:d41641dd-09ee-4029-b1ae-fb0f0a4ef62f,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-volume-6142^740a2725-ccfa-11ed-8bce-866ecf1cb72f kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-volume-6142^740a2725-ccfa-11ed-8bce-866ecf1cb72f,DevicePath:,},},Config:nil,},} Mar 27 23:53:21.543: INFO: Logging kubelet events for node i-05ec0d9f3272223ec Mar 27 23:53:21.773: INFO: Logging pods the kubelet thinks is on node i-05ec0d9f3272223ec Mar 27 23:53:21.955: INFO: pod-subpath-test-preprovisionedpv-q55n started at 2023-03-27 23:53:15 +0000 UTC (2+1 container statuses recorded) Mar 27 23:53:21.955: INFO: Init container init-volume-preprovisionedpv-q55n ready: true, restart count 0 Mar 27 23:53:21.955: INFO: Init container test-init-subpath-preprovisionedpv-q55n ready: true, restart count 0 Mar 27 23:53:21.955: INFO: Container test-container-subpath-preprovisionedpv-q55n ready: false, restart count 0 Mar 27 23:53:21.955: INFO: kube-proxy-i-05ec0d9f3272223ec started at 2023-03-27 23:25:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:21.955: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:53:21.955: INFO: hostpath-client started at 2023-03-27 23:53:04 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:21.955: INFO: Container hostpath-client ready: true, restart count 0 Mar 27 23:53:21.955: INFO: hostexec-i-05ec0d9f3272223ec-f6pkw started at 2023-03-27 23:52:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:21.955: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:53:21.955: INFO: inline-volume-tester-kwwd2 started at 2023-03-27 23:43:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:21.955: INFO: Container csi-volume-tester ready: false, restart count 0 Mar 27 23:53:21.955: INFO: ebs-csi-node-gjw52 started at 2023-03-27 23:25:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:53:21.955: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:53:21.955: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:53:21.955: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:53:21.955: INFO: coredns-559769c974-m6hh6 started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:21.955: INFO: Container coredns ready: true, restart count 0 Mar 27 23:53:21.955: INFO: labelsupdate7e586ed3-d6b2-4b24-9a28-0256a577e654 started at 2023-03-27 23:52:19 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:21.955: INFO: Container client-container ready: true, restart count 0 Mar 27 23:53:21.955: INFO: calico-node-5lq5m started at 2023-03-27 23:25:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:53:21.955: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:53:21.955: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:53:21.955: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:53:21.955: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:53:21.955: INFO: csi-hostpathplugin-0 started at 2023-03-27 23:52:35 +0000 UTC (0+7 container statuses recorded) Mar 27 23:53:21.955: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:53:21.955: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:53:21.955: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:53:21.955: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 27 23:53:21.955: INFO: Container hostpath ready: true, restart count 0 Mar 27 23:53:21.955: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:53:21.955: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:53:22.585: INFO: Latency metrics for node i-05ec0d9f3272223ec Mar 27 23:53:22.585: INFO: Logging node info for node i-0b266cca2f40af1c1 Mar 27 23:53:22.762: INFO: Node Info: &Node{ObjectMeta:{i-0b266cca2f40af1c1 5f957448-9afc-4966-b9d6-82332b7a159d 53305 0 2023-03-27 23:29:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0b266cca2f40af1c1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0b266cca2f40af1c1 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-5218":"i-0b266cca2f40af1c1","ebs.csi.aws.com":"i-0b266cca2f40af1c1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.116.128.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:30:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:30:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:53:06 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:53:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0b266cca2f40af1c1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:30:13 +0000 UTC,LastTransitionTime:2023-03-27 23:30:13 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:06 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:06 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:06 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:53:06 +0000 UTC,LastTransitionTime:2023-03-27 23:30:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.36,},NodeAddress{Type:ExternalIP,Address:13.209.22.95,},NodeAddress{Type:InternalDNS,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-209-22-95.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec247601b7b315369bfb310559a7772d,SystemUUID:ec247601-b7b3-1536-9bfb-310559a7772d,BootID:487b59c3-9394-4a90-a6ec-b20c4819fe9e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-5218^658f76bd-ccfa-11ed-8168-aa74d88430bf kubernetes.io/csi/ebs.csi.aws.com^vol-02340833e64aefd4f],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02340833e64aefd4f,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-5218^658f76bd-ccfa-11ed-8168-aa74d88430bf,DevicePath:,},},Config:nil,},} Mar 27 23:53:22.762: INFO: Logging kubelet events for node i-0b266cca2f40af1c1 Mar 27 23:53:22.948: INFO: Logging pods the kubelet thinks is on node i-0b266cca2f40af1c1 Mar 27 23:53:23.130: INFO: pod-projected-configmaps-13a5b767-9913-4450-bbfa-e347f6f38991 started at 2023-03-27 23:50:31 +0000 UTC (0+3 container statuses recorded) Mar 27 23:53:23.130: INFO: Container createcm-volume-test ready: true, restart count 0 Mar 27 23:53:23.130: INFO: Container delcm-volume-test ready: true, restart count 0 Mar 27 23:53:23.130: INFO: Container updcm-volume-test ready: true, restart count 0 Mar 27 23:53:23.130: INFO: inline-volume-tester-jh9w9 started at 2023-03-27 23:52:51 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:23.130: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 27 23:53:23.130: INFO: pfpod started at 2023-03-27 23:52:51 +0000 UTC (0+2 container statuses recorded) Mar 27 23:53:23.130: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:53:23.130: INFO: Container readiness ready: false, restart count 0 Mar 27 23:53:23.130: INFO: csi-hostpathplugin-0 started at 2023-03-27 23:52:11 +0000 UTC (0+7 container statuses recorded) Mar 27 23:53:23.130: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:53:23.130: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:53:23.130: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:53:23.130: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 27 23:53:23.130: INFO: Container hostpath ready: true, restart count 0 Mar 27 23:53:23.130: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:53:23.130: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:53:23.130: INFO: inline-volume-tester-wzxr6 started at 2023-03-27 23:52:14 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:23.130: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 27 23:53:23.130: INFO: kube-proxy-i-0b266cca2f40af1c1 started at 2023-03-27 23:29:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:23.130: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:53:23.130: INFO: calico-node-nmfrn started at 2023-03-27 23:29:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:53:23.130: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:53:23.130: INFO: Init container install-cni ready: true, restart count 1 Mar 27 23:53:23.130: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:53:23.130: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:53:23.130: INFO: ebs-csi-node-ztjm9 started at 2023-03-27 23:29:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:53:23.130: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:53:23.130: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:53:23.130: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:53:23.837: INFO: Latency metrics for node i-0b266cca2f40af1c1 Mar 27 23:53:23.837: INFO: Logging node info for node i-0d2b1184f62de758a Mar 27 23:53:24.020: INFO: Node Info: &Node{ObjectMeta:{i-0d2b1184f62de758a 2e42ccb1-9580-4126-b5d6-05a5f98403c5 45677 0 2023-03-27 23:17:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d2b1184f62de758a kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d2b1184f62de758a"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.6/19 projectcalico.org/IPv4IPIPTunnelAddr:100.120.150.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-27 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-27 23:17:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:18:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:50:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0d2b1184f62de758a,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:18:49 +0000 UTC,LastTransitionTime:2023-03-27 23:18:49 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:18:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.6,},NodeAddress{Type:ExternalIP,Address:3.35.11.126,},NodeAddress{Type:InternalDNS,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-35-11-126.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2781baf4898d15db3c853bf5b6d100,SystemUUID:ec2781ba-f489-8d15-db3c-853bf5b6d100,BootID:5e905df0-494e-4050-b6d7-400763c3dd66,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45208814,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43629012,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074727,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:53:24.020: INFO: Logging kubelet events for node i-0d2b1184f62de758a Mar 27 23:53:24.204: INFO: Logging pods the kubelet thinks is on node i-0d2b1184f62de758a Mar 27 23:53:24.388: INFO: kops-controller-w4b7t started at 2023-03-27 23:18:29 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:24.388: INFO: Container kops-controller ready: true, restart count 0 Mar 27 23:53:24.388: INFO: kube-scheduler-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:24.388: INFO: Container kube-scheduler ready: true, restart count 0 Mar 27 23:53:24.388: INFO: dns-controller-69987775c6-rzzjj started at 2023-03-27 23:18:25 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:24.388: INFO: Container dns-controller ready: true, restart count 0 Mar 27 23:53:24.388: INFO: calico-node-scbv2 started at 2023-03-27 23:18:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:53:24.388: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:53:24.388: INFO: ebs-csi-node-nvqcx started at 2023-03-27 23:18:29 +0000 UTC (0+3 container statuses recorded) Mar 27 23:53:24.388: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:53:24.388: INFO: kube-proxy-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:24.388: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:53:24.388: INFO: aws-cloud-controller-manager-776ld started at 2023-03-27 23:18:28 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:24.388: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 27 23:53:24.388: INFO: calico-kube-controllers-55f488656-vj6gh started at 2023-03-27 23:18:57 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:24.388: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 27 23:53:24.388: INFO: ebs-csi-controller-79f984545f-42vd6 started at 2023-03-27 23:19:48 +0000 UTC (0+5 container statuses recorded) Mar 27 23:53:24.388: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:53:24.388: INFO: etcd-manager-events-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:53:24.388: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:53:24.388: INFO: etcd-manager-main-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:53:24.388: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:53:24.388: INFO: kube-apiserver-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+2 container statuses recorded) Mar 27 23:53:24.388: INFO: Container healthcheck ready: true, restart count 0 Mar 27 23:53:24.388: INFO: Container kube-apiserver ready: true, restart count 3 Mar 27 23:53:24.388: INFO: kube-controller-manager-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:24.388: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 27 23:53:25.013: INFO: Latency metrics for node i-0d2b1184f62de758a Mar 27 23:53:25.013: INFO: Logging node info for node i-0e1f2cd7cf96d3020 Mar 27 23:53:25.190: INFO: Node Info: &Node{ObjectMeta:{i-0e1f2cd7cf96d3020 efcf1e66-db0e-4256-a849-b9c377eef04e 53833 0 2023-03-27 23:22:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e1f2cd7cf96d3020 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0e1f2cd7cf96d3020 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-workload-711":"i-0e1f2cd7cf96d3020","ebs.csi.aws.com":"i-0e1f2cd7cf96d3020"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.120/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.86.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:22:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:22:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:53:22 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:53:23 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0e1f2cd7cf96d3020,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:22:52 +0000 UTC,LastTransitionTime:2023-03-27 23:22:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:23 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:23 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:23 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:53:23 +0000 UTC,LastTransitionTime:2023-03-27 23:22:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.120,},NodeAddress{Type:ExternalIP,Address:15.164.224.157,},NodeAddress{Type:InternalDNS,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-164-224-157.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29484c87c97fb3d8a52788d5971660,SystemUUID:ec29484c-87c9-7fb3-d8a5-2788d5971660,BootID:b73c540f-bd49-470a-a7b3-08c31cc8c2fd,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0faa3a90c26cac81a],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0faa3a90c26cac81a,DevicePath:,},},Config:nil,},} Mar 27 23:53:25.191: INFO: Logging kubelet events for node i-0e1f2cd7cf96d3020 Mar 27 23:53:25.376: INFO: Logging pods the kubelet thinks is on node i-0e1f2cd7cf96d3020 Mar 27 23:53:25.561: INFO: ebs-csi-node-vcn4h started at 2023-03-27 23:22:28 +0000 UTC (0+3 container statuses recorded) Mar 27 23:53:25.561: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:53:25.561: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:53:25.561: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:53:25.561: INFO: kube-proxy-i-0e1f2cd7cf96d3020 started at 2023-03-27 23:22:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:25.561: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:53:25.561: INFO: calico-node-9fzr7 started at 2023-03-27 23:22:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:53:25.561: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:53:25.561: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:53:25.561: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:53:25.561: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:53:25.561: INFO: coredns-autoscaler-7cb5c5b969-2k8xf started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:25.561: INFO: Container autoscaler ready: true, restart count 0 Mar 27 23:53:25.561: INFO: pod-8a9304f3-5779-4a4c-99ac-33a0d3ce83e0 started at 2023-03-27 23:53:20 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:25.561: INFO: Container write-pod ready: false, restart count 0 Mar 27 23:53:25.561: INFO: pvc-volume-tester-znw9w started at 2023-03-27 23:52:12 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:25.561: INFO: Container volume-tester ready: true, restart count 0 Mar 27 23:53:25.561: INFO: csi-mockplugin-attacher-0 started at 2023-03-27 23:52:09 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:25.561: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:53:25.561: INFO: csi-mockplugin-0 started at 2023-03-27 23:52:09 +0000 UTC (0+3 container statuses recorded) Mar 27 23:53:25.561: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:53:25.561: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:53:25.561: INFO: Container mock ready: true, restart count 0 Mar 27 23:53:25.561: INFO: coredns-559769c974-scn7m started at 2023-03-27 23:30:58 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:25.561: INFO: Container coredns ready: true, restart count 0 Mar 27 23:53:25.561: INFO: hostexec-i-0e1f2cd7cf96d3020-tc25p started at 2023-03-27 23:53:20 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:25.561: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:53:25.561: INFO: pause-pod-1 started at 2023-03-27 23:53:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:53:25.561: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:53:26.187: INFO: Latency metrics for node i-0e1f2cd7cf96d3020 END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:53:26.187 (6.699s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/27/23 23:53:26.187 (6.699s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/27/23 23:53:26.187 STEP: Destroying namespace "provisioning-9732" for this suite. - test/e2e/framework/framework.go:351 @ 03/27/23 23:53:26.187 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/27/23 23:53:26.387 (200ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:53:26.387 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:53:26.387 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sexisting\ssingle\sfile\s\[LinuxOnly\]$'
[FAILED] failed to get logs from pod-subpath-test-dynamicpv-cqbp for test-container-subpath-dynamicpv-cqbp: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1698/pods/pod-subpath-test-dynamicpv-cqbp/log?container=test-container-subpath-dynamicpv-cqbp&previous=false": stream error: stream ID 1261; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:50:53.189from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:50:35.057 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:50:35.057 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/27/23 23:50:35.057 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/27/23 23:50:35.057 Mar 27 23:50:35.057: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:250 @ 03/27/23 23:50:35.058 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/27/23 23:50:35.598 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/27/23 23:50:35.959 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/27/23 23:50:36.312 (1.255s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:50:36.312 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:50:36.313 (0s) > Enter [It] should support existing single file [LinuxOnly] - test/e2e/storage/testsuites/subpath.go:219 @ 03/27/23 23:50:36.313 Mar 27 23:50:36.313: INFO: Creating resource for dynamic PV Mar 27 23:50:36.313: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass provisioning-1698-e2e-sczgr59 - test/e2e/storage/framework/volume_resource.go:102 @ 03/27/23 23:50:36.313 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/27/23 23:50:36.497 Mar 27 23:50:36.497: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod pod-subpath-test-dynamicpv-cqbp - test/e2e/storage/testsuites/subpath.go:691 @ 03/27/23 23:50:36.854 STEP: Creating a pod to test subpath - test/e2e/framework/pod/output/output.go:243 @ 03/27/23 23:50:36.854 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/27/23 23:50:52.468 Mar 27 23:50:52.645: INFO: Trying to get logs from node i-022003b3a45876d5d pod pod-subpath-test-dynamicpv-cqbp container test-container-subpath-dynamicpv-cqbp: <nil> Mar 27 23:50:52.821: INFO: Failed to get logs from node "i-022003b3a45876d5d" pod "pod-subpath-test-dynamicpv-cqbp" container "test-container-subpath-dynamicpv-cqbp". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1698/pods/pod-subpath-test-dynamicpv-cqbp/log?container=test-container-subpath-dynamicpv-cqbp&previous=false": stream error: stream ID 1261; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/27/23 23:50:52.821 Mar 27 23:50:53.188: INFO: Unexpected error: <*fmt.wrapError | 0xc004609b60>: failed to get logs from pod-subpath-test-dynamicpv-cqbp for test-container-subpath-dynamicpv-cqbp: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1698/pods/pod-subpath-test-dynamicpv-cqbp/log?container=test-container-subpath-dynamicpv-cqbp&previous=false": stream error: stream ID 1261; INTERNAL_ERROR; received from peer { msg: "failed to get logs from pod-subpath-test-dynamicpv-cqbp for test-container-subpath-dynamicpv-cqbp: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1698/pods/pod-subpath-test-dynamicpv-cqbp/log?container=test-container-subpath-dynamicpv-cqbp&previous=false\": stream error: stream ID 1261; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc003533740>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1698/pods/pod-subpath-test-dynamicpv-cqbp/log?container=test-container-subpath-dynamicpv-cqbp&previous=false", Err: <http2.StreamError>{ StreamID: 1261, Code: 2, Cause: <*errors.errorString | 0xc0001e7680>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from pod-subpath-test-dynamicpv-cqbp for test-container-subpath-dynamicpv-cqbp: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1698/pods/pod-subpath-test-dynamicpv-cqbp/log?container=test-container-subpath-dynamicpv-cqbp&previous=false": stream error: stream ID 1261; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:50:53.189 < Exit [It] should support existing single file [LinuxOnly] - test/e2e/storage/testsuites/subpath.go:219 @ 03/27/23 23:50:53.189 (16.876s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:50:53.189 Mar 27 23:50:53.189: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:50:53.366 (178ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:221 @ 03/27/23 23:50:53.366 STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 03/27/23 23:50:53.366 Mar 27 23:50:53.366: INFO: Deleting pod "pod-subpath-test-dynamicpv-cqbp" in namespace "provisioning-1698" STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/27/23 23:50:53.544 Mar 27 23:50:53.902: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comfmqqn" Mar 27 23:50:54.081: INFO: Waiting up to 5m0s for PersistentVolume pvc-e9f4ec1a-6292-4fcb-a0b6-cfdae8dce93e to get deleted Mar 27 23:50:54.261: INFO: PersistentVolume pvc-e9f4ec1a-6292-4fcb-a0b6-cfdae8dce93e found and phase=Released (179.633194ms) Mar 27 23:50:59.439: INFO: PersistentVolume pvc-e9f4ec1a-6292-4fcb-a0b6-cfdae8dce93e found and phase=Released (5.357921172s) Mar 27 23:51:04.623: INFO: PersistentVolume pvc-e9f4ec1a-6292-4fcb-a0b6-cfdae8dce93e found and phase=Released (10.541828038s) Mar 27 23:51:09.800: INFO: PersistentVolume pvc-e9f4ec1a-6292-4fcb-a0b6-cfdae8dce93e was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/27/23 23:51:09.8 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:221 @ 03/27/23 23:51:10.059 (16.692s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:51:10.059 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:51:10.059 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/27/23 23:51:10.059 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:51:10.059 STEP: Collecting events from namespace "provisioning-1698". - test/e2e/framework/debug/dump.go:42 @ 03/27/23 23:51:10.059 STEP: Found 12 events. - test/e2e/framework/debug/dump.go:46 @ 03/27/23 23:51:10.244 Mar 27 23:51:10.244: INFO: At 2023-03-27 23:50:36 +0000 UTC - event for ebs.csi.aws.comfmqqn: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 27 23:51:10.244: INFO: At 2023-03-27 23:50:36 +0000 UTC - event for ebs.csi.aws.comfmqqn: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 27 23:51:10.244: INFO: At 2023-03-27 23:50:36 +0000 UTC - event for ebs.csi.aws.comfmqqn: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } Provisioning: External provisioner is provisioning volume for claim "provisioning-1698/ebs.csi.aws.comfmqqn" Mar 27 23:51:10.244: INFO: At 2023-03-27 23:50:40 +0000 UTC - event for ebs.csi.aws.comfmqqn: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } ProvisioningSucceeded: Successfully provisioned volume pvc-e9f4ec1a-6292-4fcb-a0b6-cfdae8dce93e Mar 27 23:51:10.244: INFO: At 2023-03-27 23:50:40 +0000 UTC - event for pod-subpath-test-dynamicpv-cqbp: {default-scheduler } Scheduled: Successfully assigned provisioning-1698/pod-subpath-test-dynamicpv-cqbp to i-022003b3a45876d5d Mar 27 23:51:10.244: INFO: At 2023-03-27 23:50:43 +0000 UTC - event for pod-subpath-test-dynamicpv-cqbp: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-e9f4ec1a-6292-4fcb-a0b6-cfdae8dce93e" Mar 27 23:51:10.244: INFO: At 2023-03-27 23:50:47 +0000 UTC - event for pod-subpath-test-dynamicpv-cqbp: {kubelet i-022003b3a45876d5d} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 27 23:51:10.244: INFO: At 2023-03-27 23:50:47 +0000 UTC - event for pod-subpath-test-dynamicpv-cqbp: {kubelet i-022003b3a45876d5d} Created: Created container init-volume-dynamicpv-cqbp Mar 27 23:51:10.244: INFO: At 2023-03-27 23:50:47 +0000 UTC - event for pod-subpath-test-dynamicpv-cqbp: {kubelet i-022003b3a45876d5d} Started: Started container init-volume-dynamicpv-cqbp Mar 27 23:51:10.244: INFO: At 2023-03-27 23:50:48 +0000 UTC - event for pod-subpath-test-dynamicpv-cqbp: {kubelet i-022003b3a45876d5d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 27 23:51:10.244: INFO: At 2023-03-27 23:50:48 +0000 UTC - event for pod-subpath-test-dynamicpv-cqbp: {kubelet i-022003b3a45876d5d} Created: Created container test-container-subpath-dynamicpv-cqbp Mar 27 23:51:10.244: INFO: At 2023-03-27 23:50:48 +0000 UTC - event for pod-subpath-test-dynamicpv-cqbp: {kubelet i-022003b3a45876d5d} Started: Started container test-container-subpath-dynamicpv-cqbp Mar 27 23:51:10.424: INFO: POD NODE PHASE GRACE CONDITIONS Mar 27 23:51:10.424: INFO: Mar 27 23:51:10.604: INFO: Logging node info for node i-022003b3a45876d5d Mar 27 23:51:10.783: INFO: Node Info: &Node{ObjectMeta:{i-022003b3a45876d5d 20348b58-b610-41a0-accb-ca195a540336 45790 0 2023-03-27 23:33:26 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-022003b3a45876d5d kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-022003b3a45876d5d topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-9472":"i-022003b3a45876d5d","ebs.csi.aws.com":"i-022003b3a45876d5d"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.49.118/19 projectcalico.org/IPv4IPIPTunnelAddr:100.115.192.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:33:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:33:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:50:56 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-022003b3a45876d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:33:51 +0000 UTC,LastTransitionTime:2023-03-27 23:33:51 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:56 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:56 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:56 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:50:56 +0000 UTC,LastTransitionTime:2023-03-27 23:33:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.118,},NodeAddress{Type:ExternalIP,Address:54.180.103.89,},NodeAddress{Type:InternalDNS,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-103-89.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccb51411bbbb92946da93b9a452fe,SystemUUID:ec2ccb51-411b-bbb9-2946-da93b9a452fe,BootID:841d91a9-bfa5-4e0f-9279-7c83efeef8ca,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:51:10.783: INFO: Logging kubelet events for node i-022003b3a45876d5d Mar 27 23:51:10.973: INFO: Logging pods the kubelet thinks is on node i-022003b3a45876d5d Mar 27 23:51:11.164: INFO: kube-proxy-i-022003b3a45876d5d started at 2023-03-27 23:33:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:11.164: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:51:11.164: INFO: ebs-csi-node-2knd4 started at 2023-03-27 23:33:27 +0000 UTC (0+3 container statuses recorded) Mar 27 23:51:11.164: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:51:11.164: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:51:11.164: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:51:11.164: INFO: labelsupdate18826d69-79ed-419a-a047-07c13c56ec34 started at 2023-03-27 23:49:45 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:11.164: INFO: Container client-container ready: true, restart count 0 Mar 27 23:51:11.164: INFO: startup-script started at 2023-03-27 23:50:16 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:11.164: INFO: Container startup-script ready: true, restart count 0 Mar 27 23:51:11.164: INFO: inline-volume-tester2-5q2gg started at 2023-03-27 23:50:47 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:11.164: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 27 23:51:11.164: INFO: service-headless-cnsfl started at 2023-03-27 23:50:35 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:11.164: INFO: Container service-headless ready: true, restart count 0 Mar 27 23:51:11.164: INFO: csi-hostpathplugin-0 started at 2023-03-27 23:50:42 +0000 UTC (0+7 container statuses recorded) Mar 27 23:51:11.164: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:51:11.164: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:51:11.164: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:51:11.164: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 27 23:51:11.164: INFO: Container hostpath ready: true, restart count 0 Mar 27 23:51:11.164: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:51:11.164: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:51:11.164: INFO: hostexec-i-022003b3a45876d5d-6xlfg started at 2023-03-27 23:51:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:11.164: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:51:11.164: INFO: pod-e8d043f9-0661-456c-96c1-56cdd2271f2b started at 2023-03-27 23:51:06 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:11.164: INFO: Container write-pod ready: true, restart count 0 Mar 27 23:51:11.164: INFO: inline-volume-tester-chvcz started at 2023-03-27 23:50:42 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:11.164: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 27 23:51:11.164: INFO: calico-node-2wjdb started at 2023-03-27 23:33:27 +0000 UTC (3+1 container statuses recorded) Mar 27 23:51:11.164: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:51:11.164: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:51:11.164: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:51:11.164: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:51:11.164: INFO: service-headless-toggled-zdq5j started at 2023-03-27 23:50:39 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:11.164: INFO: Container service-headless-toggled ready: true, restart count 0 Mar 27 23:51:11.807: INFO: Latency metrics for node i-022003b3a45876d5d Mar 27 23:51:11.807: INFO: Logging node info for node i-05ec0d9f3272223ec Mar 27 23:51:11.984: INFO: Node Info: &Node{ObjectMeta:{i-05ec0d9f3272223ec 365deda4-148e-4593-abdf-3103630fef28 46272 0 2023-03-27 23:25:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-05ec0d9f3272223ec kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-05ec0d9f3272223ec topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1654":"i-05ec0d9f3272223ec","csi-hostpath-ephemeral-9522":"i-05ec0d9f3272223ec","ebs.csi.aws.com":"i-05ec0d9f3272223ec"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.52.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.113.211.128 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:25:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:25:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:49:17 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:51:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-05ec0d9f3272223ec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:25:42 +0000 UTC,LastTransitionTime:2023-03-27 23:25:42 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:49:17 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:49:17 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:49:17 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:49:17 +0000 UTC,LastTransitionTime:2023-03-27 23:25:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.36,},NodeAddress{Type:ExternalIP,Address:3.34.125.198,},NodeAddress{Type:InternalDNS,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-34-125-198.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dea8f8b5e44e8fd6ef34a7ee9507f,SystemUUID:ec2dea8f-8b5e-44e8-fd6e-f34a7ee9507f,BootID:d41641dd-09ee-4029-b1ae-fb0f0a4ef62f,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d,DevicePath:,},},Config:nil,},} Mar 27 23:51:11.985: INFO: Logging kubelet events for node i-05ec0d9f3272223ec Mar 27 23:51:12.169: INFO: Logging pods the kubelet thinks is on node i-05ec0d9f3272223ec Mar 27 23:51:12.353: INFO: pod-with-poststart-http-hook started at 2023-03-27 23:49:00 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:12.353: INFO: Container pod-with-poststart-http-hook ready: true, restart count 0 Mar 27 23:51:12.353: INFO: kube-proxy-i-05ec0d9f3272223ec started at 2023-03-27 23:25:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:12.353: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:51:12.353: INFO: test-grpc-3f4e7a39-e8bd-4296-bd56-c61876e85beb started at 2023-03-27 23:48:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:12.353: INFO: Container agnhost ready: true, restart count 0 Mar 27 23:51:12.353: INFO: inline-volume-tester-rtrqh started at 2023-03-27 23:50:41 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:12.353: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 27 23:51:12.353: INFO: ebs-csi-node-gjw52 started at 2023-03-27 23:25:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:51:12.353: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:51:12.353: INFO: coredns-559769c974-m6hh6 started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:12.353: INFO: Container coredns ready: true, restart count 0 Mar 27 23:51:12.353: INFO: inline-volume-tester-kwwd2 started at 2023-03-27 23:43:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:12.353: INFO: Container csi-volume-tester ready: false, restart count 0 Mar 27 23:51:12.353: INFO: test-webserver-d61a8cc2-59e4-4516-aa86-d6b94f858278 started at 2023-03-27 23:47:23 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:12.353: INFO: Container test-webserver ready: true, restart count 0 Mar 27 23:51:12.353: INFO: csi-hostpathplugin-0 started at 2023-03-27 23:51:09 +0000 UTC (0+7 container statuses recorded) Mar 27 23:51:12.353: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container hostpath ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:51:12.353: INFO: inline-volume-tester-thn46 started at 2023-03-27 23:51:09 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:12.353: INFO: Container csi-volume-tester ready: false, restart count 0 Mar 27 23:51:12.353: INFO: calico-node-5lq5m started at 2023-03-27 23:25:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:51:12.353: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:51:12.353: INFO: service-headless-p2285 started at 2023-03-27 23:50:35 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:12.353: INFO: Container service-headless ready: true, restart count 0 Mar 27 23:51:12.353: INFO: csi-hostpathplugin-0 started at 2023-03-27 23:50:41 +0000 UTC (0+7 container statuses recorded) Mar 27 23:51:12.353: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container hostpath ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:51:12.353: INFO: pod-handle-http-request started at 2023-03-27 23:48:57 +0000 UTC (0+2 container statuses recorded) Mar 27 23:51:12.353: INFO: Container container-handle-http-request ready: true, restart count 0 Mar 27 23:51:12.353: INFO: Container container-handle-https-request ready: true, restart count 0 Mar 27 23:51:12.353: INFO: service-headless-toggled-5jv9c started at 2023-03-27 23:50:39 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:12.353: INFO: Container service-headless-toggled ready: true, restart count 0 Mar 27 23:51:13.025: INFO: Latency metrics for node i-05ec0d9f3272223ec Mar 27 23:51:13.025: INFO: Logging node info for node i-0b266cca2f40af1c1 Mar 27 23:51:13.202: INFO: Node Info: &Node{ObjectMeta:{i-0b266cca2f40af1c1 5f957448-9afc-4966-b9d6-82332b7a159d 45721 0 2023-03-27 23:29:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0b266cca2f40af1c1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0b266cca2f40af1c1 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0b266cca2f40af1c1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.116.128.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:30:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:30:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:50:49 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:50:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0b266cca2f40af1c1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:30:13 +0000 UTC,LastTransitionTime:2023-03-27 23:30:13 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:53 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:53 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:53 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:50:53 +0000 UTC,LastTransitionTime:2023-03-27 23:30:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.36,},NodeAddress{Type:ExternalIP,Address:13.209.22.95,},NodeAddress{Type:InternalDNS,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-209-22-95.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec247601b7b315369bfb310559a7772d,SystemUUID:ec247601-b7b3-1536-9bfb-310559a7772d,BootID:487b59c3-9394-4a90-a6ec-b20c4819fe9e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-06b0ee226dd88e97e],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-06b0ee226dd88e97e,DevicePath:,},},Config:nil,},} Mar 27 23:51:13.202: INFO: Logging kubelet events for node i-0b266cca2f40af1c1 Mar 27 23:51:13.386: INFO: Logging pods the kubelet thinks is on node i-0b266cca2f40af1c1 Mar 27 23:51:13.576: INFO: pod-projected-configmaps-13a5b767-9913-4450-bbfa-e347f6f38991 started at 2023-03-27 23:50:31 +0000 UTC (0+3 container statuses recorded) Mar 27 23:51:13.576: INFO: Container createcm-volume-test ready: true, restart count 0 Mar 27 23:51:13.576: INFO: Container delcm-volume-test ready: true, restart count 0 Mar 27 23:51:13.576: INFO: Container updcm-volume-test ready: true, restart count 0 Mar 27 23:51:13.576: INFO: inline-volume-tester-rqqtg started at 2023-03-27 23:50:47 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:13.576: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 27 23:51:13.576: INFO: service-headless-nnbzm started at 2023-03-27 23:50:35 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:13.576: INFO: Container service-headless ready: true, restart count 0 Mar 27 23:51:13.576: INFO: service-headless-toggled-sgb6g started at 2023-03-27 23:50:39 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:13.576: INFO: Container service-headless-toggled ready: true, restart count 0 Mar 27 23:51:13.576: INFO: kube-proxy-i-0b266cca2f40af1c1 started at 2023-03-27 23:29:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:13.576: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:51:13.576: INFO: calico-node-nmfrn started at 2023-03-27 23:29:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:51:13.576: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:51:13.576: INFO: Init container install-cni ready: true, restart count 1 Mar 27 23:51:13.576: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:51:13.576: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:51:13.576: INFO: boom-server started at 2023-03-27 23:50:11 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:13.576: INFO: Container boom-server ready: true, restart count 0 Mar 27 23:51:13.576: INFO: hostexec-i-0b266cca2f40af1c1-2p6tq started at 2023-03-27 23:50:50 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:13.576: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:51:13.576: INFO: hostexec-i-0b266cca2f40af1c1-wbvls started at 2023-03-27 23:51:12 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:13.576: INFO: Container agnhost-container ready: false, restart count 0 Mar 27 23:51:13.576: INFO: ebs-csi-node-ztjm9 started at 2023-03-27 23:29:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:51:13.576: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:51:13.576: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:51:13.576: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:51:13.576: INFO: pfpod started at 2023-03-27 23:50:11 +0000 UTC (0+2 container statuses recorded) Mar 27 23:51:13.576: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:51:13.576: INFO: Container readiness ready: false, restart count 0 Mar 27 23:51:13.576: INFO: busybox-readonly-fs95a19e50-5cef-4e59-8e66-aa79c28512cd started at 2023-03-27 23:49:55 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:13.576: INFO: Container busybox-readonly-fs95a19e50-5cef-4e59-8e66-aa79c28512cd ready: true, restart count 0 Mar 27 23:51:14.216: INFO: Latency metrics for node i-0b266cca2f40af1c1 Mar 27 23:51:14.216: INFO: Logging node info for node i-0d2b1184f62de758a Mar 27 23:51:14.393: INFO: Node Info: &Node{ObjectMeta:{i-0d2b1184f62de758a 2e42ccb1-9580-4126-b5d6-05a5f98403c5 45677 0 2023-03-27 23:17:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d2b1184f62de758a kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d2b1184f62de758a"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.6/19 projectcalico.org/IPv4IPIPTunnelAddr:100.120.150.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-27 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-27 23:17:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:18:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:50:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0d2b1184f62de758a,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:18:49 +0000 UTC,LastTransitionTime:2023-03-27 23:18:49 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:18:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.6,},NodeAddress{Type:ExternalIP,Address:3.35.11.126,},NodeAddress{Type:InternalDNS,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-35-11-126.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2781baf4898d15db3c853bf5b6d100,SystemUUID:ec2781ba-f489-8d15-db3c-853bf5b6d100,BootID:5e905df0-494e-4050-b6d7-400763c3dd66,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45208814,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43629012,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074727,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:51:14.394: INFO: Logging kubelet events for node i-0d2b1184f62de758a Mar 27 23:51:14.579: INFO: Logging pods the kubelet thinks is on node i-0d2b1184f62de758a Mar 27 23:51:14.763: INFO: ebs-csi-node-nvqcx started at 2023-03-27 23:18:29 +0000 UTC (0+3 container statuses recorded) Mar 27 23:51:14.763: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:51:14.763: INFO: kops-controller-w4b7t started at 2023-03-27 23:18:29 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:14.763: INFO: Container kops-controller ready: true, restart count 0 Mar 27 23:51:14.763: INFO: kube-scheduler-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:14.763: INFO: Container kube-scheduler ready: true, restart count 0 Mar 27 23:51:14.763: INFO: dns-controller-69987775c6-rzzjj started at 2023-03-27 23:18:25 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:14.763: INFO: Container dns-controller ready: true, restart count 0 Mar 27 23:51:14.763: INFO: calico-node-scbv2 started at 2023-03-27 23:18:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:51:14.763: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:51:14.763: INFO: kube-controller-manager-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:14.763: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 27 23:51:14.763: INFO: kube-proxy-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:14.763: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:51:14.763: INFO: aws-cloud-controller-manager-776ld started at 2023-03-27 23:18:28 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:14.763: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 27 23:51:14.763: INFO: calico-kube-controllers-55f488656-vj6gh started at 2023-03-27 23:18:57 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:14.763: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 27 23:51:14.763: INFO: ebs-csi-controller-79f984545f-42vd6 started at 2023-03-27 23:19:48 +0000 UTC (0+5 container statuses recorded) Mar 27 23:51:14.763: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:51:14.763: INFO: etcd-manager-events-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:51:14.763: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:51:14.763: INFO: etcd-manager-main-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:51:14.763: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:51:14.763: INFO: kube-apiserver-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+2 container statuses recorded) Mar 27 23:51:14.763: INFO: Container healthcheck ready: true, restart count 0 Mar 27 23:51:14.763: INFO: Container kube-apiserver ready: true, restart count 3 Mar 27 23:51:15.392: INFO: Latency metrics for node i-0d2b1184f62de758a Mar 27 23:51:15.392: INFO: Logging node info for node i-0e1f2cd7cf96d3020 Mar 27 23:51:15.571: INFO: Node Info: &Node{ObjectMeta:{i-0e1f2cd7cf96d3020 efcf1e66-db0e-4256-a849-b9c377eef04e 46184 0 2023-03-27 23:22:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e1f2cd7cf96d3020 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0e1f2cd7cf96d3020 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e1f2cd7cf96d3020"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.120/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.86.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:22:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:22:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:51:09 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0e1f2cd7cf96d3020,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:22:52 +0000 UTC,LastTransitionTime:2023-03-27 23:22:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:19 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:19 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:19 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:50:19 +0000 UTC,LastTransitionTime:2023-03-27 23:22:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.120,},NodeAddress{Type:ExternalIP,Address:15.164.224.157,},NodeAddress{Type:InternalDNS,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-164-224-157.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29484c87c97fb3d8a52788d5971660,SystemUUID:ec29484c-87c9-7fb3-d8a5-2788d5971660,BootID:b73c540f-bd49-470a-a7b3-08c31cc8c2fd,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:51:15.572: INFO: Logging kubelet events for node i-0e1f2cd7cf96d3020 Mar 27 23:51:15.759: INFO: Logging pods the kubelet thinks is on node i-0e1f2cd7cf96d3020 Mar 27 23:51:15.945: INFO: coredns-autoscaler-7cb5c5b969-2k8xf started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:15.945: INFO: Container autoscaler ready: true, restart count 0 Mar 27 23:51:15.945: INFO: csi-mockplugin-0 started at 2023-03-27 23:50:26 +0000 UTC (0+4 container statuses recorded) Mar 27 23:51:15.945: INFO: Container busybox ready: true, restart count 0 Mar 27 23:51:15.945: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:51:15.945: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:51:15.945: INFO: Container mock ready: true, restart count 0 Mar 27 23:51:15.945: INFO: pod-terminate-status-0-2 started at 2023-03-27 23:51:13 +0000 UTC (1+1 container statuses recorded) Mar 27 23:51:15.945: INFO: Init container fail ready: false, restart count 0 Mar 27 23:51:15.945: INFO: Container blocked ready: false, restart count 0 Mar 27 23:51:15.945: INFO: pod-terminate-status-2-2 started at 2023-03-27 23:51:14 +0000 UTC (1+1 container statuses recorded) Mar 27 23:51:15.945: INFO: Init container fail ready: false, restart count 0 Mar 27 23:51:15.945: INFO: Container blocked ready: false, restart count 0 Mar 27 23:51:15.945: INFO: pod-terminate-status-1-2 started at 2023-03-27 23:51:14 +0000 UTC (1+1 container statuses recorded) Mar 27 23:51:15.945: INFO: Init container fail ready: false, restart count 0 Mar 27 23:51:15.945: INFO: Container blocked ready: false, restart count 0 Mar 27 23:51:15.945: INFO: coredns-559769c974-scn7m started at 2023-03-27 23:30:58 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:15.945: INFO: Container coredns ready: true, restart count 0 Mar 27 23:51:15.945: INFO: all-pods-removed-cjbrb started at 2023-03-27 23:50:53 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:15.945: INFO: Container c ready: true, restart count 0 Mar 27 23:51:15.945: INFO: verify-service-up-exec-pod-mhr5z started at 2023-03-27 23:51:09 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:15.945: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:51:15.945: INFO: pod-secrets-7ed2cc3c-9cc5-44fb-9638-f4367534a2b7 started at 2023-03-27 23:51:11 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:15.945: INFO: Container secret-volume-test ready: false, restart count 0 Mar 27 23:51:15.945: INFO: annotationupdate875ee3a4-f332-4685-9c2a-a0c81252d4cc started at 2023-03-27 23:51:03 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:15.945: INFO: Container client-container ready: true, restart count 0 Mar 27 23:51:15.945: INFO: ebs-csi-node-vcn4h started at 2023-03-27 23:22:28 +0000 UTC (0+3 container statuses recorded) Mar 27 23:51:15.945: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:51:15.945: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:51:15.945: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:51:15.945: INFO: all-pods-removed-88pvc started at 2023-03-27 23:50:53 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:15.945: INFO: Container c ready: true, restart count 0 Mar 27 23:51:15.945: INFO: kube-proxy-i-0e1f2cd7cf96d3020 started at 2023-03-27 23:22:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:15.945: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:51:15.945: INFO: calico-node-9fzr7 started at 2023-03-27 23:22:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:51:15.945: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:51:15.945: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:51:15.945: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:51:15.945: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:51:15.945: INFO: verify-service-up-host-exec-pod started at 2023-03-27 23:51:06 +0000 UTC (0+1 container statuses recorded) Mar 27 23:51:15.945: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:51:16.648: INFO: Latency metrics for node i-0e1f2cd7cf96d3020 END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:51:16.648 (6.589s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/27/23 23:51:16.648 (6.589s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/27/23 23:51:16.648 STEP: Destroying namespace "provisioning-1698" for this suite. - test/e2e/framework/framework.go:351 @ 03/27/23 23:51:16.648 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/27/23 23:51:16.826 (178ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:51:16.826 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:51:16.826 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sfile\sas\ssubpath\s\[LinuxOnly\]$'
[FAILED] failed to get logs from pod-subpath-test-dynamicpv-ngj9 for test-container-subpath-dynamicpv-ngj9: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4512/pods/pod-subpath-test-dynamicpv-ngj9/log?container=test-container-subpath-dynamicpv-ngj9&previous=false": stream error: stream ID 2513; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:54:50.43from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:54:10.489 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:54:10.489 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/27/23 23:54:10.489 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/27/23 23:54:10.489 Mar 27 23:54:10.489: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:250 @ 03/27/23 23:54:10.49 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/27/23 23:54:11.026 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/27/23 23:54:11.383 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/27/23 23:54:11.736 (1.247s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:54:11.736 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:54:11.736 (0s) > Enter [It] should support file as subpath [LinuxOnly] - test/e2e/storage/testsuites/subpath.go:230 @ 03/27/23 23:54:11.736 Mar 27 23:54:11.736: INFO: Creating resource for dynamic PV Mar 27 23:54:11.736: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass provisioning-4512-e2e-scgfszw - test/e2e/storage/framework/volume_resource.go:102 @ 03/27/23 23:54:11.737 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/27/23 23:54:11.915 Mar 27 23:54:11.915: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod pod-subpath-test-dynamicpv-ngj9 - test/e2e/storage/testsuites/subpath.go:489 @ 03/27/23 23:54:12.288 STEP: Creating a pod to test atomic-volume-subpath - test/e2e/framework/pod/output/output.go:243 @ 03/27/23 23:54:12.288 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/27/23 23:54:49.708 Mar 27 23:54:49.888: INFO: Trying to get logs from node i-05ec0d9f3272223ec pod pod-subpath-test-dynamicpv-ngj9 container test-container-subpath-dynamicpv-ngj9: <nil> Mar 27 23:54:50.067: INFO: Failed to get logs from node "i-05ec0d9f3272223ec" pod "pod-subpath-test-dynamicpv-ngj9" container "test-container-subpath-dynamicpv-ngj9". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4512/pods/pod-subpath-test-dynamicpv-ngj9/log?container=test-container-subpath-dynamicpv-ngj9&previous=false": stream error: stream ID 2513; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/27/23 23:54:50.067 Mar 27 23:54:50.429: INFO: Unexpected error: <*fmt.wrapError | 0xc003f0cf20>: failed to get logs from pod-subpath-test-dynamicpv-ngj9 for test-container-subpath-dynamicpv-ngj9: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4512/pods/pod-subpath-test-dynamicpv-ngj9/log?container=test-container-subpath-dynamicpv-ngj9&previous=false": stream error: stream ID 2513; INTERNAL_ERROR; received from peer { msg: "failed to get logs from pod-subpath-test-dynamicpv-ngj9 for test-container-subpath-dynamicpv-ngj9: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4512/pods/pod-subpath-test-dynamicpv-ngj9/log?container=test-container-subpath-dynamicpv-ngj9&previous=false\": stream error: stream ID 2513; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc003f76b70>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4512/pods/pod-subpath-test-dynamicpv-ngj9/log?container=test-container-subpath-dynamicpv-ngj9&previous=false", Err: <http2.StreamError>{ StreamID: 2513, Code: 2, Cause: <*errors.errorString | 0xc0001b3650>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from pod-subpath-test-dynamicpv-ngj9 for test-container-subpath-dynamicpv-ngj9: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-4512/pods/pod-subpath-test-dynamicpv-ngj9/log?container=test-container-subpath-dynamicpv-ngj9&previous=false": stream error: stream ID 2513; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:54:50.43 < Exit [It] should support file as subpath [LinuxOnly] - test/e2e/storage/testsuites/subpath.go:230 @ 03/27/23 23:54:50.43 (38.693s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:54:50.43 Mar 27 23:54:50.430: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:54:50.608 (178ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:232 @ 03/27/23 23:54:50.608 STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 03/27/23 23:54:50.608 Mar 27 23:54:50.608: INFO: Deleting pod "pod-subpath-test-dynamicpv-ngj9" in namespace "provisioning-4512" STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/27/23 23:54:50.785 Mar 27 23:54:51.140: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.com6gzgl" Mar 27 23:54:51.319: INFO: Waiting up to 5m0s for PersistentVolume pvc-d8911f3b-aae4-4f60-aefb-ae19abd1dd4c to get deleted Mar 27 23:54:51.497: INFO: PersistentVolume pvc-d8911f3b-aae4-4f60-aefb-ae19abd1dd4c found and phase=Released (178.684369ms) Mar 27 23:54:56.681: INFO: PersistentVolume pvc-d8911f3b-aae4-4f60-aefb-ae19abd1dd4c found and phase=Released (5.361979499s) Mar 27 23:55:01.859: INFO: PersistentVolume pvc-d8911f3b-aae4-4f60-aefb-ae19abd1dd4c found and phase=Released (10.540478154s) Mar 27 23:55:07.037: INFO: PersistentVolume pvc-d8911f3b-aae4-4f60-aefb-ae19abd1dd4c was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/27/23 23:55:07.037 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:232 @ 03/27/23 23:55:07.217 (16.609s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:55:07.217 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:55:07.217 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/27/23 23:55:07.217 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:55:07.217 STEP: Collecting events from namespace "provisioning-4512". - test/e2e/framework/debug/dump.go:42 @ 03/27/23 23:55:07.217 STEP: Found 12 events. - test/e2e/framework/debug/dump.go:46 @ 03/27/23 23:55:07.395 Mar 27 23:55:07.395: INFO: At 2023-03-27 23:54:12 +0000 UTC - event for ebs.csi.aws.com6gzgl: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 27 23:55:07.395: INFO: At 2023-03-27 23:54:12 +0000 UTC - event for ebs.csi.aws.com6gzgl: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 27 23:55:07.395: INFO: At 2023-03-27 23:54:12 +0000 UTC - event for ebs.csi.aws.com6gzgl: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } Provisioning: External provisioner is provisioning volume for claim "provisioning-4512/ebs.csi.aws.com6gzgl" Mar 27 23:55:07.395: INFO: At 2023-03-27 23:54:15 +0000 UTC - event for ebs.csi.aws.com6gzgl: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } ProvisioningSucceeded: Successfully provisioned volume pvc-d8911f3b-aae4-4f60-aefb-ae19abd1dd4c Mar 27 23:55:07.395: INFO: At 2023-03-27 23:54:16 +0000 UTC - event for pod-subpath-test-dynamicpv-ngj9: {default-scheduler } Scheduled: Successfully assigned provisioning-4512/pod-subpath-test-dynamicpv-ngj9 to i-05ec0d9f3272223ec Mar 27 23:55:07.395: INFO: At 2023-03-27 23:54:18 +0000 UTC - event for pod-subpath-test-dynamicpv-ngj9: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-d8911f3b-aae4-4f60-aefb-ae19abd1dd4c" Mar 27 23:55:07.395: INFO: At 2023-03-27 23:54:25 +0000 UTC - event for pod-subpath-test-dynamicpv-ngj9: {kubelet i-05ec0d9f3272223ec} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 27 23:55:07.395: INFO: At 2023-03-27 23:54:25 +0000 UTC - event for pod-subpath-test-dynamicpv-ngj9: {kubelet i-05ec0d9f3272223ec} Created: Created container init-volume-dynamicpv-ngj9 Mar 27 23:55:07.395: INFO: At 2023-03-27 23:54:25 +0000 UTC - event for pod-subpath-test-dynamicpv-ngj9: {kubelet i-05ec0d9f3272223ec} Started: Started container init-volume-dynamicpv-ngj9 Mar 27 23:55:07.395: INFO: At 2023-03-27 23:54:26 +0000 UTC - event for pod-subpath-test-dynamicpv-ngj9: {kubelet i-05ec0d9f3272223ec} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 27 23:55:07.395: INFO: At 2023-03-27 23:54:26 +0000 UTC - event for pod-subpath-test-dynamicpv-ngj9: {kubelet i-05ec0d9f3272223ec} Created: Created container test-container-subpath-dynamicpv-ngj9 Mar 27 23:55:07.395: INFO: At 2023-03-27 23:54:26 +0000 UTC - event for pod-subpath-test-dynamicpv-ngj9: {kubelet i-05ec0d9f3272223ec} Started: Started container test-container-subpath-dynamicpv-ngj9 Mar 27 23:55:07.576: INFO: POD NODE PHASE GRACE CONDITIONS Mar 27 23:55:07.576: INFO: Mar 27 23:55:07.756: INFO: Logging node info for node i-022003b3a45876d5d Mar 27 23:55:07.936: INFO: Node Info: &Node{ObjectMeta:{i-022003b3a45876d5d 20348b58-b610-41a0-accb-ca195a540336 56891 0 2023-03-27 23:33:26 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-022003b3a45876d5d kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-022003b3a45876d5d topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-022003b3a45876d5d"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.49.118/19 projectcalico.org/IPv4IPIPTunnelAddr:100.115.192.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:33:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:33:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:54:48 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:54:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-022003b3a45876d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:33:51 +0000 UTC,LastTransitionTime:2023-03-27 23:33:51 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:50 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:50 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:50 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:54:50 +0000 UTC,LastTransitionTime:2023-03-27 23:33:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.118,},NodeAddress{Type:ExternalIP,Address:54.180.103.89,},NodeAddress{Type:InternalDNS,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-103-89.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccb51411bbbb92946da93b9a452fe,SystemUUID:ec2ccb51-411b-bbb9-2946-da93b9a452fe,BootID:841d91a9-bfa5-4e0f-9279-7c83efeef8ca,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-027c7267daa9fbf92],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-027c7267daa9fbf92,DevicePath:,},},Config:nil,},} Mar 27 23:55:07.936: INFO: Logging kubelet events for node i-022003b3a45876d5d Mar 27 23:55:08.121: INFO: Logging pods the kubelet thinks is on node i-022003b3a45876d5d Mar 27 23:55:08.308: INFO: pod2 started at 2023-03-27 23:55:07 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:08.308: INFO: Container agnhost-container ready: false, restart count 0 Mar 27 23:55:08.308: INFO: up-down-3-k947m started at 2023-03-27 23:55:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:08.308: INFO: Container up-down-3 ready: true, restart count 0 Mar 27 23:55:08.308: INFO: webserver-pod started at 2023-03-27 23:52:55 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:08.308: INFO: Container agnhost ready: false, restart count 0 Mar 27 23:55:08.308: INFO: hostexec-i-022003b3a45876d5d-4p4t4 started at 2023-03-27 23:55:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:08.308: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:08.308: INFO: calico-node-2wjdb started at 2023-03-27 23:33:27 +0000 UTC (3+1 container statuses recorded) Mar 27 23:55:08.308: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:55:08.308: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:55:08.308: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:55:08.308: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:55:08.308: INFO: netserver-0 started at 2023-03-27 23:54:44 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:08.308: INFO: Container webserver ready: true, restart count 0 Mar 27 23:55:08.308: INFO: ebs-csi-node-2knd4 started at 2023-03-27 23:33:27 +0000 UTC (0+3 container statuses recorded) Mar 27 23:55:08.308: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:55:08.308: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:55:08.308: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:55:08.308: INFO: pfpod started at 2023-03-27 23:52:53 +0000 UTC (0+2 container statuses recorded) Mar 27 23:55:08.308: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:55:08.308: INFO: Container readiness ready: false, restart count 0 Mar 27 23:55:08.308: INFO: kube-proxy-i-022003b3a45876d5d started at 2023-03-27 23:33:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:08.308: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:55:08.308: INFO: ss-0 started at 2023-03-27 23:54:46 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:08.308: INFO: Container webserver ready: true, restart count 0 Mar 27 23:55:08.308: INFO: hostexec-i-022003b3a45876d5d-wp6rd started at 2023-03-27 23:55:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:08.308: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:08.308: INFO: pfpod started at 2023-03-27 23:53:13 +0000 UTC (0+2 container statuses recorded) Mar 27 23:55:08.308: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:55:08.308: INFO: Container readiness ready: false, restart count 0 Mar 27 23:55:08.308: INFO: up-down-2-jwgr8 started at 2023-03-27 23:54:19 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:08.308: INFO: Container up-down-2 ready: true, restart count 0 Mar 27 23:55:09.022: INFO: Latency metrics for node i-022003b3a45876d5d Mar 27 23:55:09.022: INFO: Logging node info for node i-05ec0d9f3272223ec Mar 27 23:55:09.200: INFO: Node Info: &Node{ObjectMeta:{i-05ec0d9f3272223ec 365deda4-148e-4593-abdf-3103630fef28 57431 0 2023-03-27 23:25:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-05ec0d9f3272223ec kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-05ec0d9f3272223ec topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-05ec0d9f3272223ec"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.52.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.113.211.128 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:25:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:25:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:54:59 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:55:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-05ec0d9f3272223ec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:25:42 +0000 UTC,LastTransitionTime:2023-03-27 23:25:42 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:55:05 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:55:05 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:55:05 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:55:05 +0000 UTC,LastTransitionTime:2023-03-27 23:25:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.36,},NodeAddress{Type:ExternalIP,Address:3.34.125.198,},NodeAddress{Type:InternalDNS,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-34-125-198.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dea8f8b5e44e8fd6ef34a7ee9507f,SystemUUID:ec2dea8f-8b5e-44e8-fd6e-f34a7ee9507f,BootID:d41641dd-09ee-4029-b1ae-fb0f0a4ef62f,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d kubernetes.io/csi/ebs.csi.aws.com^vol-054fcfb0606bc608a],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-054fcfb0606bc608a,DevicePath:,},},Config:nil,},} Mar 27 23:55:09.200: INFO: Logging kubelet events for node i-05ec0d9f3272223ec Mar 27 23:55:09.403: INFO: Logging pods the kubelet thinks is on node i-05ec0d9f3272223ec Mar 27 23:55:09.588: INFO: coredns-559769c974-m6hh6 started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:09.588: INFO: Container coredns ready: true, restart count 0 Mar 27 23:55:09.588: INFO: inline-volume-tester-kwwd2 started at 2023-03-27 23:43:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:09.588: INFO: Container csi-volume-tester ready: false, restart count 0 Mar 27 23:55:09.588: INFO: pod-ephm-test-secret-jtp6 started at 2023-03-27 23:53:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:09.588: INFO: Container test-container-subpath-secret-jtp6 ready: false, restart count 0 Mar 27 23:55:09.588: INFO: ebs-csi-node-gjw52 started at 2023-03-27 23:25:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:55:09.588: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:55:09.588: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:55:09.588: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:55:09.588: INFO: up-down-3-kzgf9 started at 2023-03-27 23:55:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:09.588: INFO: Container up-down-3 ready: true, restart count 0 Mar 27 23:55:09.588: INFO: simpletest.deployment-5d9bdff859-shqbm started at 2023-03-27 23:55:03 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:09.588: INFO: Container nginx ready: true, restart count 0 Mar 27 23:55:09.588: INFO: test-container-pod started at 2023-03-27 23:55:08 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:09.588: INFO: Container webserver ready: false, restart count 0 Mar 27 23:55:09.588: INFO: pod1 started at 2023-03-27 23:54:57 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:09.588: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:09.588: INFO: calico-node-5lq5m started at 2023-03-27 23:25:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:55:09.588: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:55:09.588: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:55:09.588: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:55:09.588: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:55:09.588: INFO: probe-test-044ed117-c53a-4e96-8fd7-3e37a6cefce1 started at 2023-03-27 23:55:06 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:09.588: INFO: Container probe-test-044ed117-c53a-4e96-8fd7-3e37a6cefce1 ready: false, restart count 0 Mar 27 23:55:09.588: INFO: ss-1 started at 2023-03-27 23:54:56 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:09.588: INFO: Container webserver ready: true, restart count 0 Mar 27 23:55:09.588: INFO: httpd started at 2023-03-27 23:54:23 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:09.588: INFO: Container httpd ready: true, restart count 0 Mar 27 23:55:09.588: INFO: up-down-2-jsqpk started at 2023-03-27 23:54:19 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:09.588: INFO: Container up-down-2 ready: true, restart count 0 Mar 27 23:55:09.588: INFO: netserver-1 started at 2023-03-27 23:54:44 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:09.588: INFO: Container webserver ready: true, restart count 0 Mar 27 23:55:09.588: INFO: verify-service-up-host-exec-pod started at 2023-03-27 23:55:05 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:09.588: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:09.588: INFO: kube-proxy-i-05ec0d9f3272223ec started at 2023-03-27 23:25:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:09.588: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:55:10.234: INFO: Latency metrics for node i-05ec0d9f3272223ec Mar 27 23:55:10.234: INFO: Logging node info for node i-0b266cca2f40af1c1 Mar 27 23:55:10.412: INFO: Node Info: &Node{ObjectMeta:{i-0b266cca2f40af1c1 5f957448-9afc-4966-b9d6-82332b7a159d 57146 0 2023-03-27 23:29:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0b266cca2f40af1c1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0b266cca2f40af1c1 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7886":"i-0b266cca2f40af1c1","csi-mock-csi-mock-volumes-attach-8822":"i-0b266cca2f40af1c1","csi-mock-csi-mock-volumes-expansion-2991":"i-0b266cca2f40af1c1","ebs.csi.aws.com":"i-0b266cca2f40af1c1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.116.128.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:30:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:30:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:54:58 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:54:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0b266cca2f40af1c1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:30:13 +0000 UTC,LastTransitionTime:2023-03-27 23:30:13 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:58 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:58 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:58 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:54:58 +0000 UTC,LastTransitionTime:2023-03-27 23:30:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.36,},NodeAddress{Type:ExternalIP,Address:13.209.22.95,},NodeAddress{Type:InternalDNS,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-209-22-95.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec247601b7b315369bfb310559a7772d,SystemUUID:ec247601-b7b3-1536-9bfb-310559a7772d,BootID:487b59c3-9394-4a90-a6ec-b20c4819fe9e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7886^c04b2cb6-ccfa-11ed-a1b1-fa34c4e6c5a5 kubernetes.io/csi/csi-hostpath-ephemeral-7886^c4676d58-ccfa-11ed-a1b1-fa34c4e6c5a5 kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-2991^9ab3bf37-ccfa-11ed-a28d-ead8e583275f],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-2991^9ab3bf37-ccfa-11ed-a28d-ead8e583275f,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7886^c04b2cb6-ccfa-11ed-a1b1-fa34c4e6c5a5,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7886^c4676d58-ccfa-11ed-a1b1-fa34c4e6c5a5,DevicePath:,},},Config:nil,},} Mar 27 23:55:10.412: INFO: Logging kubelet events for node i-0b266cca2f40af1c1 Mar 27 23:55:10.609: INFO: Logging pods the kubelet thinks is on node i-0b266cca2f40af1c1 Mar 27 23:55:10.792: INFO: kube-proxy-i-0b266cca2f40af1c1 started at 2023-03-27 23:29:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:10.792: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:55:10.792: INFO: calico-node-nmfrn started at 2023-03-27 23:29:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:55:10.792: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:55:10.792: INFO: Init container install-cni ready: true, restart count 1 Mar 27 23:55:10.792: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:55:10.792: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:55:10.792: INFO: inline-volume-tester-8gmqj started at 2023-03-27 23:54:46 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:10.792: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 27 23:55:10.792: INFO: csi-mockplugin-0 started at 2023-03-27 23:54:51 +0000 UTC (0+3 container statuses recorded) Mar 27 23:55:10.792: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:55:10.792: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:55:10.792: INFO: Container mock ready: true, restart count 0 Mar 27 23:55:10.792: INFO: csi-mockplugin-0 started at 2023-03-27 23:53:39 +0000 UTC (0+3 container statuses recorded) Mar 27 23:55:10.792: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:55:10.792: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:55:10.792: INFO: Container mock ready: true, restart count 0 Mar 27 23:55:10.792: INFO: csi-mockplugin-resizer-0 started at 2023-03-27 23:53:40 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:10.792: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:55:10.792: INFO: csi-mockplugin-attacher-0 started at 2023-03-27 23:53:39 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:10.792: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:55:10.792: INFO: pvc-volume-tester-kjswl started at 2023-03-27 23:53:43 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:10.792: INFO: Container volume-tester ready: true, restart count 0 Mar 27 23:55:10.792: INFO: csi-hostpathplugin-0 started at 2023-03-27 23:54:43 +0000 UTC (0+7 container statuses recorded) Mar 27 23:55:10.792: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:55:10.792: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:55:10.792: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:55:10.792: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 27 23:55:10.792: INFO: Container hostpath ready: true, restart count 0 Mar 27 23:55:10.792: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:55:10.792: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:55:10.792: INFO: hostexec-i-0b266cca2f40af1c1-xcn9p started at 2023-03-27 23:54:47 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:10.792: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:10.792: INFO: ebs-csi-node-ztjm9 started at 2023-03-27 23:29:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:55:10.792: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:55:10.792: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:55:10.792: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:55:10.792: INFO: inline-volume-tester2-4j4zs started at 2023-03-27 23:54:53 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:10.792: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 27 23:55:10.792: INFO: netserver-2 started at 2023-03-27 23:54:44 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:10.792: INFO: Container webserver ready: true, restart count 0 Mar 27 23:55:10.792: INFO: hostexec-i-0b266cca2f40af1c1-cvflq started at 2023-03-27 23:55:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:10.792: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:10.792: INFO: pfpod started at 2023-03-27 23:52:51 +0000 UTC (0+2 container statuses recorded) Mar 27 23:55:10.792: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:55:10.792: INFO: Container readiness ready: false, restart count 0 Mar 27 23:55:10.792: INFO: up-down-2-w7vfg started at 2023-03-27 23:54:19 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:10.792: INFO: Container up-down-2 ready: true, restart count 0 Mar 27 23:55:11.488: INFO: Latency metrics for node i-0b266cca2f40af1c1 Mar 27 23:55:11.488: INFO: Logging node info for node i-0d2b1184f62de758a Mar 27 23:55:11.666: INFO: Node Info: &Node{ObjectMeta:{i-0d2b1184f62de758a 2e42ccb1-9580-4126-b5d6-05a5f98403c5 45677 0 2023-03-27 23:17:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d2b1184f62de758a kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d2b1184f62de758a"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.6/19 projectcalico.org/IPv4IPIPTunnelAddr:100.120.150.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-27 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-27 23:17:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:18:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:50:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0d2b1184f62de758a,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:18:49 +0000 UTC,LastTransitionTime:2023-03-27 23:18:49 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:18:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.6,},NodeAddress{Type:ExternalIP,Address:3.35.11.126,},NodeAddress{Type:InternalDNS,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-35-11-126.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2781baf4898d15db3c853bf5b6d100,SystemUUID:ec2781ba-f489-8d15-db3c-853bf5b6d100,BootID:5e905df0-494e-4050-b6d7-400763c3dd66,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45208814,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43629012,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074727,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:55:11.667: INFO: Logging kubelet events for node i-0d2b1184f62de758a Mar 27 23:55:11.851: INFO: Logging pods the kubelet thinks is on node i-0d2b1184f62de758a Mar 27 23:55:12.038: INFO: etcd-manager-main-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:55:12.038: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:55:12.038: INFO: kube-apiserver-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+2 container statuses recorded) Mar 27 23:55:12.038: INFO: Container healthcheck ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Container kube-apiserver ready: true, restart count 3 Mar 27 23:55:12.038: INFO: kube-controller-manager-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:12.038: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 27 23:55:12.038: INFO: kube-proxy-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:12.038: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:55:12.038: INFO: aws-cloud-controller-manager-776ld started at 2023-03-27 23:18:28 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:12.038: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 27 23:55:12.038: INFO: calico-kube-controllers-55f488656-vj6gh started at 2023-03-27 23:18:57 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:12.038: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 27 23:55:12.038: INFO: ebs-csi-controller-79f984545f-42vd6 started at 2023-03-27 23:19:48 +0000 UTC (0+5 container statuses recorded) Mar 27 23:55:12.038: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:55:12.038: INFO: etcd-manager-events-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:55:12.038: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:55:12.038: INFO: dns-controller-69987775c6-rzzjj started at 2023-03-27 23:18:25 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:12.038: INFO: Container dns-controller ready: true, restart count 0 Mar 27 23:55:12.038: INFO: calico-node-scbv2 started at 2023-03-27 23:18:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:55:12.038: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:55:12.038: INFO: ebs-csi-node-nvqcx started at 2023-03-27 23:18:29 +0000 UTC (0+3 container statuses recorded) Mar 27 23:55:12.038: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:55:12.038: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:55:12.038: INFO: kops-controller-w4b7t started at 2023-03-27 23:18:29 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:12.038: INFO: Container kops-controller ready: true, restart count 0 Mar 27 23:55:12.038: INFO: kube-scheduler-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:12.038: INFO: Container kube-scheduler ready: true, restart count 0 Mar 27 23:55:12.647: INFO: Latency metrics for node i-0d2b1184f62de758a Mar 27 23:55:12.647: INFO: Logging node info for node i-0e1f2cd7cf96d3020 Mar 27 23:55:12.830: INFO: Node Info: &Node{ObjectMeta:{i-0e1f2cd7cf96d3020 efcf1e66-db0e-4256-a849-b9c377eef04e 57242 0 2023-03-27 23:22:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e1f2cd7cf96d3020 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0e1f2cd7cf96d3020 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e1f2cd7cf96d3020"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.120/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.86.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:22:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:22:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:55:00 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0e1f2cd7cf96d3020,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:22:52 +0000 UTC,LastTransitionTime:2023-03-27 23:22:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:35 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:35 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:35 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:54:35 +0000 UTC,LastTransitionTime:2023-03-27 23:22:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.120,},NodeAddress{Type:ExternalIP,Address:15.164.224.157,},NodeAddress{Type:InternalDNS,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-164-224-157.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29484c87c97fb3d8a52788d5971660,SystemUUID:ec29484c-87c9-7fb3-d8a5-2788d5971660,BootID:b73c540f-bd49-470a-a7b3-08c31cc8c2fd,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:55:12.830: INFO: Logging kubelet events for node i-0e1f2cd7cf96d3020 Mar 27 23:55:13.016: INFO: Logging pods the kubelet thinks is on node i-0e1f2cd7cf96d3020 Mar 27 23:55:13.202: INFO: kube-proxy-i-0e1f2cd7cf96d3020 started at 2023-03-27 23:22:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.202: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:55:13.202: INFO: calico-node-9fzr7 started at 2023-03-27 23:22:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:55:13.202: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:55:13.202: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:55:13.202: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:55:13.202: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:55:13.202: INFO: host-test-container-pod started at 2023-03-27 23:55:08 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.202: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:13.202: INFO: coredns-autoscaler-7cb5c5b969-2k8xf started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.202: INFO: Container autoscaler ready: true, restart count 0 Mar 27 23:55:13.202: INFO: ss-2 started at 2023-03-27 23:55:10 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.202: INFO: Container webserver ready: false, restart count 0 Mar 27 23:55:13.202: INFO: hostexec-i-0e1f2cd7cf96d3020-h74tq started at 2023-03-27 23:55:04 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.202: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:13.202: INFO: pod-csi-inline-volumes started at 2023-03-27 23:54:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.202: INFO: Container pod-csi-inline-volumes ready: false, restart count 0 Mar 27 23:55:13.202: INFO: up-down-3-r5hn8 started at 2023-03-27 23:55:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.202: INFO: Container up-down-3 ready: true, restart count 0 Mar 27 23:55:13.202: INFO: coredns-559769c974-scn7m started at 2023-03-27 23:30:58 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.202: INFO: Container coredns ready: true, restart count 0 Mar 27 23:55:13.202: INFO: hostexec-i-0e1f2cd7cf96d3020-sj6dd started at 2023-03-27 23:54:32 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.202: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:13.202: INFO: execpodqt86v started at 2023-03-27 23:55:00 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.202: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:13.202: INFO: explicit-root-uid started at 2023-03-27 23:55:06 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.202: INFO: Container explicit-root-uid ready: false, restart count 0 Mar 27 23:55:13.202: INFO: ebs-csi-node-vcn4h started at 2023-03-27 23:22:28 +0000 UTC (0+3 container statuses recorded) Mar 27 23:55:13.202: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:55:13.202: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:55:13.202: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:55:13.202: INFO: netserver-3 started at 2023-03-27 23:54:44 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.202: INFO: Container webserver ready: true, restart count 0 Mar 27 23:55:13.841: INFO: Latency metrics for node i-0e1f2cd7cf96d3020 END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:55:13.841 (6.624s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/27/23 23:55:13.841 (6.624s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/27/23 23:55:13.841 STEP: Destroying namespace "provisioning-4512" for this suite. - test/e2e/framework/framework.go:351 @ 03/27/23 23:55:13.841 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/27/23 23:55:14.02 (180ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:55:14.021 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:55:14.021 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\snon\-existent\spath$'
[FAILED] failed to get logs from pod-subpath-test-dynamicpv-pphd for test-container-volume-dynamicpv-pphd: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5039/pods/pod-subpath-test-dynamicpv-pphd/log?container=test-container-volume-dynamicpv-pphd&previous=false": stream error: stream ID 449; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:44:51.444from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:44:31.167 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:44:31.167 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/27/23 23:44:31.167 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/27/23 23:44:31.167 Mar 27 23:44:31.167: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:250 @ 03/27/23 23:44:31.168 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/27/23 23:44:31.7 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/27/23 23:44:32.056 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/27/23 23:44:32.408 (1.241s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:44:32.408 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:44:32.408 (0s) > Enter [It] should support non-existent path - test/e2e/storage/testsuites/subpath.go:194 @ 03/27/23 23:44:32.408 Mar 27 23:44:32.408: INFO: Creating resource for dynamic PV Mar 27 23:44:32.409: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass provisioning-5039-e2e-scdqcj4 - test/e2e/storage/framework/volume_resource.go:102 @ 03/27/23 23:44:32.409 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/27/23 23:44:32.586 Mar 27 23:44:32.586: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod pod-subpath-test-dynamicpv-pphd - test/e2e/storage/testsuites/subpath.go:691 @ 03/27/23 23:44:32.943 STEP: Creating a pod to test subpath - test/e2e/framework/pod/output/output.go:243 @ 03/27/23 23:44:32.943 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/27/23 23:44:50.724 Mar 27 23:44:50.901: INFO: Trying to get logs from node i-0b266cca2f40af1c1 pod pod-subpath-test-dynamicpv-pphd container test-container-volume-dynamicpv-pphd: <nil> Mar 27 23:44:51.079: INFO: Failed to get logs from node "i-0b266cca2f40af1c1" pod "pod-subpath-test-dynamicpv-pphd" container "test-container-volume-dynamicpv-pphd". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5039/pods/pod-subpath-test-dynamicpv-pphd/log?container=test-container-volume-dynamicpv-pphd&previous=false": stream error: stream ID 449; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/27/23 23:44:51.079 Mar 27 23:44:51.444: INFO: Unexpected error: <*fmt.wrapError | 0xc007272d40>: failed to get logs from pod-subpath-test-dynamicpv-pphd for test-container-volume-dynamicpv-pphd: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5039/pods/pod-subpath-test-dynamicpv-pphd/log?container=test-container-volume-dynamicpv-pphd&previous=false": stream error: stream ID 449; INTERNAL_ERROR; received from peer { msg: "failed to get logs from pod-subpath-test-dynamicpv-pphd for test-container-volume-dynamicpv-pphd: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5039/pods/pod-subpath-test-dynamicpv-pphd/log?container=test-container-volume-dynamicpv-pphd&previous=false\": stream error: stream ID 449; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc0028d4210>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5039/pods/pod-subpath-test-dynamicpv-pphd/log?container=test-container-volume-dynamicpv-pphd&previous=false", Err: <http2.StreamError>{ StreamID: 449, Code: 2, Cause: <*errors.errorString | 0xc00022b670>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from pod-subpath-test-dynamicpv-pphd for test-container-volume-dynamicpv-pphd: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-5039/pods/pod-subpath-test-dynamicpv-pphd/log?container=test-container-volume-dynamicpv-pphd&previous=false": stream error: stream ID 449; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:44:51.444 < Exit [It] should support non-existent path - test/e2e/storage/testsuites/subpath.go:194 @ 03/27/23 23:44:51.444 (19.035s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:44:51.444 Mar 27 23:44:51.444: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:44:51.628 (184ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:196 @ 03/27/23 23:44:51.628 STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 03/27/23 23:44:51.628 Mar 27 23:44:51.628: INFO: Deleting pod "pod-subpath-test-dynamicpv-pphd" in namespace "provisioning-5039" STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/27/23 23:44:51.978 Mar 27 23:44:52.334: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comnq247" Mar 27 23:44:52.514: INFO: Waiting up to 5m0s for PersistentVolume pvc-7d377364-fb81-46e7-9a06-8760c04b8545 to get deleted Mar 27 23:44:52.691: INFO: PersistentVolume pvc-7d377364-fb81-46e7-9a06-8760c04b8545 found and phase=Released (176.898815ms) Mar 27 23:44:57.868: INFO: PersistentVolume pvc-7d377364-fb81-46e7-9a06-8760c04b8545 found and phase=Released (5.353725701s) Mar 27 23:45:03.045: INFO: PersistentVolume pvc-7d377364-fb81-46e7-9a06-8760c04b8545 found and phase=Released (10.53078338s) Mar 27 23:45:08.222: INFO: PersistentVolume pvc-7d377364-fb81-46e7-9a06-8760c04b8545 found and phase=Released (15.707873373s) Mar 27 23:45:13.399: INFO: PersistentVolume pvc-7d377364-fb81-46e7-9a06-8760c04b8545 found and phase=Released (20.88441334s) Mar 27 23:45:18.576: INFO: PersistentVolume pvc-7d377364-fb81-46e7-9a06-8760c04b8545 found and phase=Released (26.061762308s) Mar 27 23:45:23.757: INFO: PersistentVolume pvc-7d377364-fb81-46e7-9a06-8760c04b8545 was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/27/23 23:45:23.757 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:196 @ 03/27/23 23:45:23.934 (32.307s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:45:23.934 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:45:23.935 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/27/23 23:45:23.935 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:45:23.935 STEP: Collecting events from namespace "provisioning-5039". - test/e2e/framework/debug/dump.go:42 @ 03/27/23 23:45:23.935 STEP: Found 15 events. - test/e2e/framework/debug/dump.go:46 @ 03/27/23 23:45:24.112 Mar 27 23:45:24.112: INFO: At 2023-03-27 23:44:32 +0000 UTC - event for ebs.csi.aws.comnq247: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 27 23:45:24.112: INFO: At 2023-03-27 23:44:33 +0000 UTC - event for ebs.csi.aws.comnq247: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 27 23:45:24.112: INFO: At 2023-03-27 23:44:33 +0000 UTC - event for ebs.csi.aws.comnq247: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } Provisioning: External provisioner is provisioning volume for claim "provisioning-5039/ebs.csi.aws.comnq247" Mar 27 23:45:24.112: INFO: At 2023-03-27 23:44:36 +0000 UTC - event for ebs.csi.aws.comnq247: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } ProvisioningSucceeded: Successfully provisioned volume pvc-7d377364-fb81-46e7-9a06-8760c04b8545 Mar 27 23:45:24.112: INFO: At 2023-03-27 23:44:37 +0000 UTC - event for pod-subpath-test-dynamicpv-pphd: {default-scheduler } Scheduled: Successfully assigned provisioning-5039/pod-subpath-test-dynamicpv-pphd to i-0b266cca2f40af1c1 Mar 27 23:45:24.112: INFO: At 2023-03-27 23:44:39 +0000 UTC - event for pod-subpath-test-dynamicpv-pphd: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-7d377364-fb81-46e7-9a06-8760c04b8545" Mar 27 23:45:24.112: INFO: At 2023-03-27 23:44:46 +0000 UTC - event for pod-subpath-test-dynamicpv-pphd: {kubelet i-0b266cca2f40af1c1} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 27 23:45:24.112: INFO: At 2023-03-27 23:44:46 +0000 UTC - event for pod-subpath-test-dynamicpv-pphd: {kubelet i-0b266cca2f40af1c1} Created: Created container test-init-subpath-dynamicpv-pphd Mar 27 23:45:24.112: INFO: At 2023-03-27 23:44:46 +0000 UTC - event for pod-subpath-test-dynamicpv-pphd: {kubelet i-0b266cca2f40af1c1} Started: Started container test-init-subpath-dynamicpv-pphd Mar 27 23:45:24.112: INFO: At 2023-03-27 23:44:46 +0000 UTC - event for pod-subpath-test-dynamicpv-pphd: {kubelet i-0b266cca2f40af1c1} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 27 23:45:24.112: INFO: At 2023-03-27 23:44:46 +0000 UTC - event for pod-subpath-test-dynamicpv-pphd: {kubelet i-0b266cca2f40af1c1} Created: Created container test-container-subpath-dynamicpv-pphd Mar 27 23:45:24.112: INFO: At 2023-03-27 23:44:46 +0000 UTC - event for pod-subpath-test-dynamicpv-pphd: {kubelet i-0b266cca2f40af1c1} Started: Started container test-container-subpath-dynamicpv-pphd Mar 27 23:45:24.112: INFO: At 2023-03-27 23:44:46 +0000 UTC - event for pod-subpath-test-dynamicpv-pphd: {kubelet i-0b266cca2f40af1c1} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 27 23:45:24.112: INFO: At 2023-03-27 23:44:46 +0000 UTC - event for pod-subpath-test-dynamicpv-pphd: {kubelet i-0b266cca2f40af1c1} Created: Created container test-container-volume-dynamicpv-pphd Mar 27 23:45:24.112: INFO: At 2023-03-27 23:44:47 +0000 UTC - event for pod-subpath-test-dynamicpv-pphd: {kubelet i-0b266cca2f40af1c1} Started: Started container test-container-volume-dynamicpv-pphd Mar 27 23:45:24.291: INFO: POD NODE PHASE GRACE CONDITIONS Mar 27 23:45:24.291: INFO: Mar 27 23:45:24.473: INFO: Logging node info for node i-022003b3a45876d5d Mar 27 23:45:24.650: INFO: Node Info: &Node{ObjectMeta:{i-022003b3a45876d5d 20348b58-b610-41a0-accb-ca195a540336 29695 0 2023-03-27 23:33:26 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-022003b3a45876d5d kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-022003b3a45876d5d topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-022003b3a45876d5d"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.49.118/19 projectcalico.org/IPv4IPIPTunnelAddr:100.115.192.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:33:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:33:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:45:09 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-022003b3a45876d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:33:51 +0000 UTC,LastTransitionTime:2023-03-27 23:33:51 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:45:09 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:45:09 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:45:09 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:45:09 +0000 UTC,LastTransitionTime:2023-03-27 23:33:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.118,},NodeAddress{Type:ExternalIP,Address:54.180.103.89,},NodeAddress{Type:InternalDNS,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-103-89.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccb51411bbbb92946da93b9a452fe,SystemUUID:ec2ccb51-411b-bbb9-2946-da93b9a452fe,BootID:841d91a9-bfa5-4e0f-9279-7c83efeef8ca,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:45:24.651: INFO: Logging kubelet events for node i-022003b3a45876d5d Mar 27 23:45:24.836: INFO: Logging pods the kubelet thinks is on node i-022003b3a45876d5d Mar 27 23:45:25.021: INFO: calico-node-2wjdb started at 2023-03-27 23:33:27 +0000 UTC (3+1 container statuses recorded) Mar 27 23:45:25.021: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:45:25.021: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:45:25.021: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:45:25.021: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:45:25.021: INFO: pod-secrets-f9689818-53f9-4e56-b6f9-b3f9fe8d517b started at 2023-03-27 23:43:23 +0000 UTC (0+3 container statuses recorded) Mar 27 23:45:25.021: INFO: Container creates-volume-test ready: true, restart count 0 Mar 27 23:45:25.021: INFO: Container dels-volume-test ready: true, restart count 0 Mar 27 23:45:25.021: INFO: Container upds-volume-test ready: true, restart count 0 Mar 27 23:45:25.021: INFO: test-rolling-update-with-lb-66756c6d5c-76dfj started at 2023-03-27 23:44:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:25.021: INFO: Container agnhost ready: true, restart count 0 Mar 27 23:45:25.021: INFO: run-test started at 2023-03-27 23:45:16 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:25.021: INFO: Container run-test ready: false, restart count 0 Mar 27 23:45:25.021: INFO: netserver-0 started at 2023-03-27 23:43:55 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:25.021: INFO: Container webserver ready: true, restart count 0 Mar 27 23:45:25.021: INFO: kube-proxy-i-022003b3a45876d5d started at 2023-03-27 23:33:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:25.021: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:45:25.021: INFO: ebs-csi-node-2knd4 started at 2023-03-27 23:33:27 +0000 UTC (0+3 container statuses recorded) Mar 27 23:45:25.021: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:45:25.021: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:45:25.021: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:45:25.021: INFO: pod-projected-secrets-dd6baa8b-98ae-4958-ba9e-c1b4ab77fec8 started at 2023-03-27 23:42:41 +0000 UTC (0+3 container statuses recorded) Mar 27 23:45:25.021: INFO: Container creates-volume-test ready: true, restart count 0 Mar 27 23:45:25.021: INFO: Container dels-volume-test ready: true, restart count 0 Mar 27 23:45:25.021: INFO: Container upds-volume-test ready: true, restart count 0 Mar 27 23:45:25.021: INFO: httpd started at 2023-03-27 23:45:05 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:25.021: INFO: Container httpd ready: true, restart count 0 Mar 27 23:45:25.021: INFO: liveness-59c03831-9f09-46ff-8a16-864e9b532e0a started at 2023-03-27 23:43:32 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:25.021: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:45:25.632: INFO: Latency metrics for node i-022003b3a45876d5d Mar 27 23:45:25.633: INFO: Logging node info for node i-05ec0d9f3272223ec Mar 27 23:45:25.813: INFO: Node Info: &Node{ObjectMeta:{i-05ec0d9f3272223ec 365deda4-148e-4593-abdf-3103630fef28 29729 0 2023-03-27 23:25:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-05ec0d9f3272223ec kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-05ec0d9f3272223ec topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-05ec0d9f3272223ec"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.52.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.113.211.128 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:25:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:25:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:44:42 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:45:10 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-05ec0d9f3272223ec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:25:42 +0000 UTC,LastTransitionTime:2023-03-27 23:25:42 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:44:41 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:44:41 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:44:41 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:44:41 +0000 UTC,LastTransitionTime:2023-03-27 23:25:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.36,},NodeAddress{Type:ExternalIP,Address:3.34.125.198,},NodeAddress{Type:InternalDNS,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-34-125-198.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dea8f8b5e44e8fd6ef34a7ee9507f,SystemUUID:ec2dea8f-8b5e-44e8-fd6e-f34a7ee9507f,BootID:d41641dd-09ee-4029-b1ae-fb0f0a4ef62f,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d kubernetes.io/csi/ebs.csi.aws.com^vol-07047db3747027362],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-07047db3747027362,DevicePath:,},},Config:nil,},} Mar 27 23:45:25.814: INFO: Logging kubelet events for node i-05ec0d9f3272223ec Mar 27 23:45:26.000: INFO: Logging pods the kubelet thinks is on node i-05ec0d9f3272223ec Mar 27 23:45:26.183: INFO: kube-proxy-i-05ec0d9f3272223ec started at 2023-03-27 23:25:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:26.183: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:45:26.183: INFO: inline-volume-tester-kwwd2 started at 2023-03-27 23:43:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:26.183: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 27 23:45:26.183: INFO: pfpod started at 2023-03-27 23:44:02 +0000 UTC (0+2 container statuses recorded) Mar 27 23:45:26.183: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:45:26.183: INFO: Container readiness ready: false, restart count 0 Mar 27 23:45:26.183: INFO: ebs-csi-node-gjw52 started at 2023-03-27 23:25:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:45:26.183: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:45:26.183: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:45:26.183: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:45:26.183: INFO: coredns-559769c974-m6hh6 started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:26.183: INFO: Container coredns ready: true, restart count 0 Mar 27 23:45:26.183: INFO: terminate-cmd-rpa7a1223e8-6c01-4e34-8d55-18968d60eb66 started at 2023-03-27 23:45:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:26.183: INFO: Container terminate-cmd-rpa ready: false, restart count 1 Mar 27 23:45:26.183: INFO: pod-init-e70304ed-e3a9-4ec0-be09-073ded9b3720 started at 2023-03-27 23:45:21 +0000 UTC (2+1 container statuses recorded) Mar 27 23:45:26.183: INFO: Init container init1 ready: true, restart count 0 Mar 27 23:45:26.183: INFO: Init container init2 ready: true, restart count 0 Mar 27 23:45:26.183: INFO: Container run1 ready: false, restart count 0 Mar 27 23:45:26.183: INFO: calico-node-5lq5m started at 2023-03-27 23:25:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:45:26.183: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:45:26.183: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:45:26.183: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:45:26.183: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:45:26.183: INFO: busybox-readonly-true-0b63ef98-c29e-4a9b-bfea-07857e364a99 started at 2023-03-27 23:45:23 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:26.183: INFO: Container busybox-readonly-true-0b63ef98-c29e-4a9b-bfea-07857e364a99 ready: false, restart count 0 Mar 27 23:45:26.183: INFO: annotationupdatea4e73dcd-e891-4a87-9946-dfe5c6a0f816 started at 2023-03-27 23:44:01 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:26.183: INFO: Container client-container ready: true, restart count 0 Mar 27 23:45:26.183: INFO: netserver-1 started at 2023-03-27 23:43:56 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:26.183: INFO: Container webserver ready: true, restart count 0 Mar 27 23:45:26.823: INFO: Latency metrics for node i-05ec0d9f3272223ec Mar 27 23:45:26.823: INFO: Logging node info for node i-0b266cca2f40af1c1 Mar 27 23:45:27.000: INFO: Node Info: &Node{ObjectMeta:{i-0b266cca2f40af1c1 5f957448-9afc-4966-b9d6-82332b7a159d 30169 0 2023-03-27 23:29:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0b266cca2f40af1c1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0b266cca2f40af1c1 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0b266cca2f40af1c1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.116.128.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:30:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:30:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:45:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0b266cca2f40af1c1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:30:13 +0000 UTC,LastTransitionTime:2023-03-27 23:30:13 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:45:26 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:45:26 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:45:26 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:45:26 +0000 UTC,LastTransitionTime:2023-03-27 23:30:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.36,},NodeAddress{Type:ExternalIP,Address:13.209.22.95,},NodeAddress{Type:InternalDNS,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-209-22-95.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec247601b7b315369bfb310559a7772d,SystemUUID:ec247601-b7b3-1536-9bfb-310559a7772d,BootID:487b59c3-9394-4a90-a6ec-b20c4819fe9e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:45:27.000: INFO: Logging kubelet events for node i-0b266cca2f40af1c1 Mar 27 23:45:27.187: INFO: Logging pods the kubelet thinks is on node i-0b266cca2f40af1c1 Mar 27 23:45:27.370: INFO: ebs-csi-node-ztjm9 started at 2023-03-27 23:29:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:45:27.370: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:45:27.370: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:45:27.370: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:45:27.370: INFO: test-container-pod started at 2023-03-27 23:44:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:27.370: INFO: Container webserver ready: true, restart count 0 Mar 27 23:45:27.370: INFO: pod-configmaps-cb64ad83-3af4-4c45-9bea-66eb9f5eeb11 started at 2023-03-27 23:44:25 +0000 UTC (0+3 container statuses recorded) Mar 27 23:45:27.370: INFO: Container createcm-volume-test ready: true, restart count 0 Mar 27 23:45:27.370: INFO: Container delcm-volume-test ready: true, restart count 0 Mar 27 23:45:27.370: INFO: Container updcm-volume-test ready: true, restart count 0 Mar 27 23:45:27.370: INFO: concurrent-27999345-dkxq5 started at 2023-03-27 23:45:00 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:27.370: INFO: Container c ready: true, restart count 0 Mar 27 23:45:27.370: INFO: test-rolling-update-with-lb-66756c6d5c-bccx5 started at 2023-03-27 23:44:58 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:27.370: INFO: Container agnhost ready: true, restart count 0 Mar 27 23:45:27.370: INFO: kube-proxy-i-0b266cca2f40af1c1 started at 2023-03-27 23:29:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:27.370: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:45:27.370: INFO: calico-node-nmfrn started at 2023-03-27 23:29:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:45:27.370: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:45:27.370: INFO: Init container install-cni ready: true, restart count 1 Mar 27 23:45:27.370: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:45:27.370: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:45:27.370: INFO: hostexec-i-0b266cca2f40af1c1-v94k5 started at 2023-03-27 23:45:13 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:27.370: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:45:27.370: INFO: dns-test-2f89b074-9dc7-4009-974f-754faf19064a started at 2023-03-27 23:45:05 +0000 UTC (0+3 container statuses recorded) Mar 27 23:45:27.370: INFO: Container jessie-querier ready: true, restart count 0 Mar 27 23:45:27.370: INFO: Container querier ready: true, restart count 0 Mar 27 23:45:27.370: INFO: Container webserver ready: true, restart count 0 Mar 27 23:45:27.370: INFO: netserver-2 started at 2023-03-27 23:43:56 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:27.370: INFO: Container webserver ready: true, restart count 0 Mar 27 23:45:27.993: INFO: Latency metrics for node i-0b266cca2f40af1c1 Mar 27 23:45:27.993: INFO: Logging node info for node i-0d2b1184f62de758a Mar 27 23:45:28.170: INFO: Node Info: &Node{ObjectMeta:{i-0d2b1184f62de758a 2e42ccb1-9580-4126-b5d6-05a5f98403c5 17460 0 2023-03-27 23:17:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d2b1184f62de758a kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d2b1184f62de758a"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.6/19 projectcalico.org/IPv4IPIPTunnelAddr:100.120.150.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-27 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-27 23:17:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:18:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:40:39 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0d2b1184f62de758a,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:18:49 +0000 UTC,LastTransitionTime:2023-03-27 23:18:49 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:40:39 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:40:39 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:40:39 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:40:39 +0000 UTC,LastTransitionTime:2023-03-27 23:18:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.6,},NodeAddress{Type:ExternalIP,Address:3.35.11.126,},NodeAddress{Type:InternalDNS,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-35-11-126.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2781baf4898d15db3c853bf5b6d100,SystemUUID:ec2781ba-f489-8d15-db3c-853bf5b6d100,BootID:5e905df0-494e-4050-b6d7-400763c3dd66,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45208814,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43629012,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074727,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:45:28.170: INFO: Logging kubelet events for node i-0d2b1184f62de758a Mar 27 23:45:28.362: INFO: Logging pods the kubelet thinks is on node i-0d2b1184f62de758a Mar 27 23:45:28.551: INFO: calico-kube-controllers-55f488656-vj6gh started at 2023-03-27 23:18:57 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:28.551: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 27 23:45:28.551: INFO: ebs-csi-controller-79f984545f-42vd6 started at 2023-03-27 23:19:48 +0000 UTC (0+5 container statuses recorded) Mar 27 23:45:28.551: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:45:28.551: INFO: etcd-manager-events-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:45:28.551: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:45:28.551: INFO: etcd-manager-main-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:45:28.551: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:45:28.551: INFO: kube-apiserver-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+2 container statuses recorded) Mar 27 23:45:28.551: INFO: Container healthcheck ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Container kube-apiserver ready: true, restart count 3 Mar 27 23:45:28.551: INFO: kube-controller-manager-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:28.551: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 27 23:45:28.551: INFO: kube-proxy-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:28.551: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:45:28.551: INFO: aws-cloud-controller-manager-776ld started at 2023-03-27 23:18:28 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:28.551: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 27 23:45:28.551: INFO: kube-scheduler-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:28.551: INFO: Container kube-scheduler ready: true, restart count 0 Mar 27 23:45:28.551: INFO: dns-controller-69987775c6-rzzjj started at 2023-03-27 23:18:25 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:28.551: INFO: Container dns-controller ready: true, restart count 0 Mar 27 23:45:28.551: INFO: calico-node-scbv2 started at 2023-03-27 23:18:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:45:28.551: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:45:28.551: INFO: ebs-csi-node-nvqcx started at 2023-03-27 23:18:29 +0000 UTC (0+3 container statuses recorded) Mar 27 23:45:28.551: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:45:28.551: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:45:28.551: INFO: kops-controller-w4b7t started at 2023-03-27 23:18:29 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:28.551: INFO: Container kops-controller ready: true, restart count 0 Mar 27 23:45:29.190: INFO: Latency metrics for node i-0d2b1184f62de758a Mar 27 23:45:29.190: INFO: Logging node info for node i-0e1f2cd7cf96d3020 Mar 27 23:45:29.367: INFO: Node Info: &Node{ObjectMeta:{i-0e1f2cd7cf96d3020 efcf1e66-db0e-4256-a849-b9c377eef04e 30105 0 2023-03-27 23:22:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e1f2cd7cf96d3020 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0e1f2cd7cf96d3020 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-2708":"i-0e1f2cd7cf96d3020","csi-mock-csi-mock-volumes-expansion-4700":"i-0e1f2cd7cf96d3020","csi-mock-csi-mock-volumes-fsgroup-policy-51":"i-0e1f2cd7cf96d3020","ebs.csi.aws.com":"i-0e1f2cd7cf96d3020"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.120/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.86.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:22:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:22:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:45:24 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:45:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0e1f2cd7cf96d3020,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:22:52 +0000 UTC,LastTransitionTime:2023-03-27 23:22:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:45:24 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:45:24 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:45:24 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:45:24 +0000 UTC,LastTransitionTime:2023-03-27 23:22:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.120,},NodeAddress{Type:ExternalIP,Address:15.164.224.157,},NodeAddress{Type:InternalDNS,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-164-224-157.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29484c87c97fb3d8a52788d5971660,SystemUUID:ec29484c-87c9-7fb3-d8a5-2788d5971660,BootID:b73c540f-bd49-470a-a7b3-08c31cc8c2fd,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4700^301e3bbe-ccf9-11ed-a882-f233d05cf674],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4700^301e3bbe-ccf9-11ed-a882-f233d05cf674,DevicePath:,},},Config:nil,},} Mar 27 23:45:29.367: INFO: Logging kubelet events for node i-0e1f2cd7cf96d3020 Mar 27 23:45:29.560: INFO: Logging pods the kubelet thinks is on node i-0e1f2cd7cf96d3020 Mar 27 23:45:29.747: INFO: ebs-csi-node-vcn4h started at 2023-03-27 23:22:28 +0000 UTC (0+3 container statuses recorded) Mar 27 23:45:29.747: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:45:29.747: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:45:29.747: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:45:29.747: INFO: csi-hostpathplugin-0 started at 2023-03-27 23:43:28 +0000 UTC (0+7 container statuses recorded) Mar 27 23:45:29.747: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:45:29.747: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:45:29.747: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:45:29.747: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 27 23:45:29.747: INFO: Container hostpath ready: true, restart count 0 Mar 27 23:45:29.747: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:45:29.747: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:45:29.747: INFO: hostexec-i-0e1f2cd7cf96d3020-j4d7p started at 2023-03-27 23:44:58 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:29.747: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:45:29.747: INFO: kube-proxy-i-0e1f2cd7cf96d3020 started at 2023-03-27 23:22:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:29.747: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:45:29.747: INFO: calico-node-9fzr7 started at 2023-03-27 23:22:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:45:29.747: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:45:29.747: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:45:29.747: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:45:29.747: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:45:29.747: INFO: coredns-autoscaler-7cb5c5b969-2k8xf started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:29.747: INFO: Container autoscaler ready: true, restart count 0 Mar 27 23:45:29.747: INFO: hostexec-i-0e1f2cd7cf96d3020-dfdc6 started at 2023-03-27 23:45:24 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:29.747: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:45:29.747: INFO: csi-mockplugin-resizer-0 started at 2023-03-27 23:43:31 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:29.747: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:45:29.747: INFO: csi-mockplugin-attacher-0 started at 2023-03-27 23:43:31 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:29.747: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:45:29.747: INFO: test-rolling-update-with-lb-66756c6d5c-qn6sl started at 2023-03-27 23:45:01 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:29.747: INFO: Container agnhost ready: true, restart count 0 Mar 27 23:45:29.747: INFO: pvc-volume-tester-s4989 started at 2023-03-27 23:45:20 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:29.747: INFO: Container volume-tester ready: true, restart count 0 Mar 27 23:45:29.747: INFO: csi-mockplugin-0 started at 2023-03-27 23:43:31 +0000 UTC (0+3 container statuses recorded) Mar 27 23:45:29.747: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:45:29.747: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:45:29.747: INFO: Container mock ready: true, restart count 0 Mar 27 23:45:29.747: INFO: coredns-559769c974-scn7m started at 2023-03-27 23:30:58 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:29.747: INFO: Container coredns ready: true, restart count 0 Mar 27 23:45:29.747: INFO: csi-mockplugin-0 started at 2023-03-27 23:45:08 +0000 UTC (0+3 container statuses recorded) Mar 27 23:45:29.747: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:45:29.747: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:45:29.747: INFO: Container mock ready: true, restart count 0 Mar 27 23:45:29.747: INFO: pvc-volume-tester-cwxsw started at 2023-03-27 23:43:35 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:29.747: INFO: Container volume-tester ready: true, restart count 0 Mar 27 23:45:29.747: INFO: netserver-3 started at 2023-03-27 23:43:56 +0000 UTC (0+1 container statuses recorded) Mar 27 23:45:29.747: INFO: Container webserver ready: true, restart count 0 Mar 27 23:45:30.392: INFO: Latency metrics for node i-0e1f2cd7cf96d3020 END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:45:30.392 (6.457s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/27/23 23:45:30.392 (6.457s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/27/23 23:45:30.392 STEP: Destroying namespace "provisioning-5039" for this suite. - test/e2e/framework/framework.go:351 @ 03/27/23 23:45:30.392 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/27/23 23:45:30.574 (182ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:45:30.574 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:45:30.574 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sreadOnly\sdirectory\sspecified\sin\sthe\svolumeMount$'
[FAILED] failed to get logs from pod-subpath-test-dynamicpv-hbs7 for test-container-subpath-dynamicpv-hbs7: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1057/pods/pod-subpath-test-dynamicpv-hbs7/log?container=test-container-subpath-dynamicpv-hbs7&previous=false": stream error: stream ID 2997; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:49:46.276from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:49:23.802 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:49:23.802 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/27/23 23:49:23.802 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/27/23 23:49:23.802 Mar 27 23:49:23.802: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:250 @ 03/27/23 23:49:23.803 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/27/23 23:49:24.336 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/27/23 23:49:24.688 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/27/23 23:49:25.048 (1.246s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:49:25.048 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:49:25.048 (0s) > Enter [It] should support readOnly directory specified in the volumeMount - test/e2e/storage/testsuites/subpath.go:367 @ 03/27/23 23:49:25.048 Mar 27 23:49:25.048: INFO: Creating resource for dynamic PV Mar 27 23:49:25.048: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass provisioning-1057-e2e-scbk7fl - test/e2e/storage/framework/volume_resource.go:102 @ 03/27/23 23:49:25.049 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/27/23 23:49:25.246 Mar 27 23:49:25.246: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod pod-subpath-test-dynamicpv-hbs7 - test/e2e/storage/testsuites/subpath.go:691 @ 03/27/23 23:49:25.601 STEP: Creating a pod to test subpath - test/e2e/framework/pod/output/output.go:243 @ 03/27/23 23:49:25.601 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/27/23 23:49:45.555 Mar 27 23:49:45.731: INFO: Trying to get logs from node i-0e1f2cd7cf96d3020 pod pod-subpath-test-dynamicpv-hbs7 container test-container-subpath-dynamicpv-hbs7: <nil> Mar 27 23:49:45.908: INFO: Failed to get logs from node "i-0e1f2cd7cf96d3020" pod "pod-subpath-test-dynamicpv-hbs7" container "test-container-subpath-dynamicpv-hbs7". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1057/pods/pod-subpath-test-dynamicpv-hbs7/log?container=test-container-subpath-dynamicpv-hbs7&previous=false": stream error: stream ID 2997; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/27/23 23:49:45.908 Mar 27 23:49:46.276: INFO: Unexpected error: <*fmt.wrapError | 0xc000d62840>: failed to get logs from pod-subpath-test-dynamicpv-hbs7 for test-container-subpath-dynamicpv-hbs7: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1057/pods/pod-subpath-test-dynamicpv-hbs7/log?container=test-container-subpath-dynamicpv-hbs7&previous=false": stream error: stream ID 2997; INTERNAL_ERROR; received from peer { msg: "failed to get logs from pod-subpath-test-dynamicpv-hbs7 for test-container-subpath-dynamicpv-hbs7: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1057/pods/pod-subpath-test-dynamicpv-hbs7/log?container=test-container-subpath-dynamicpv-hbs7&previous=false\": stream error: stream ID 2997; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc004b50810>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1057/pods/pod-subpath-test-dynamicpv-hbs7/log?container=test-container-subpath-dynamicpv-hbs7&previous=false", Err: <http2.StreamError>{ StreamID: 2997, Code: 2, Cause: <*errors.errorString | 0xc00022b660>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from pod-subpath-test-dynamicpv-hbs7 for test-container-subpath-dynamicpv-hbs7: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-1057/pods/pod-subpath-test-dynamicpv-hbs7/log?container=test-container-subpath-dynamicpv-hbs7&previous=false": stream error: stream ID 2997; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:49:46.276 < Exit [It] should support readOnly directory specified in the volumeMount - test/e2e/storage/testsuites/subpath.go:367 @ 03/27/23 23:49:46.276 (21.228s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:49:46.276 Mar 27 23:49:46.276: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:49:46.472 (196ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:369 @ 03/27/23 23:49:46.472 STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 03/27/23 23:49:46.472 Mar 27 23:49:46.472: INFO: Deleting pod "pod-subpath-test-dynamicpv-hbs7" in namespace "provisioning-1057" STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/27/23 23:49:46.665 Mar 27 23:49:47.020: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comwmnrc" Mar 27 23:49:47.217: INFO: Waiting up to 5m0s for PersistentVolume pvc-0f7e16b6-3175-4554-825c-01730a5a169b to get deleted Mar 27 23:49:47.395: INFO: PersistentVolume pvc-0f7e16b6-3175-4554-825c-01730a5a169b found and phase=Released (177.898197ms) Mar 27 23:49:52.572: INFO: PersistentVolume pvc-0f7e16b6-3175-4554-825c-01730a5a169b found and phase=Released (5.355451276s) Mar 27 23:49:57.750: INFO: PersistentVolume pvc-0f7e16b6-3175-4554-825c-01730a5a169b found and phase=Released (10.532755975s) Mar 27 23:50:02.926: INFO: PersistentVolume pvc-0f7e16b6-3175-4554-825c-01730a5a169b was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/27/23 23:50:02.926 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:369 @ 03/27/23 23:50:03.106 (16.634s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:50:03.106 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:50:03.107 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/27/23 23:50:03.107 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:50:03.107 STEP: Collecting events from namespace "provisioning-1057". - test/e2e/framework/debug/dump.go:42 @ 03/27/23 23:50:03.107 STEP: Found 15 events. - test/e2e/framework/debug/dump.go:46 @ 03/27/23 23:50:03.283 Mar 27 23:50:03.283: INFO: At 2023-03-27 23:49:25 +0000 UTC - event for ebs.csi.aws.comwmnrc: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 27 23:50:03.283: INFO: At 2023-03-27 23:49:25 +0000 UTC - event for ebs.csi.aws.comwmnrc: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } Provisioning: External provisioner is provisioning volume for claim "provisioning-1057/ebs.csi.aws.comwmnrc" Mar 27 23:50:03.283: INFO: At 2023-03-27 23:49:25 +0000 UTC - event for ebs.csi.aws.comwmnrc: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 27 23:50:03.283: INFO: At 2023-03-27 23:49:29 +0000 UTC - event for ebs.csi.aws.comwmnrc: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } ProvisioningSucceeded: Successfully provisioned volume pvc-0f7e16b6-3175-4554-825c-01730a5a169b Mar 27 23:50:03.283: INFO: At 2023-03-27 23:49:29 +0000 UTC - event for pod-subpath-test-dynamicpv-hbs7: {default-scheduler } Scheduled: Successfully assigned provisioning-1057/pod-subpath-test-dynamicpv-hbs7 to i-0e1f2cd7cf96d3020 Mar 27 23:50:03.283: INFO: At 2023-03-27 23:49:32 +0000 UTC - event for pod-subpath-test-dynamicpv-hbs7: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-0f7e16b6-3175-4554-825c-01730a5a169b" Mar 27 23:50:03.283: INFO: At 2023-03-27 23:49:39 +0000 UTC - event for pod-subpath-test-dynamicpv-hbs7: {kubelet i-0e1f2cd7cf96d3020} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 27 23:50:03.283: INFO: At 2023-03-27 23:49:39 +0000 UTC - event for pod-subpath-test-dynamicpv-hbs7: {kubelet i-0e1f2cd7cf96d3020} Created: Created container init-volume-dynamicpv-hbs7 Mar 27 23:50:03.283: INFO: At 2023-03-27 23:49:40 +0000 UTC - event for pod-subpath-test-dynamicpv-hbs7: {kubelet i-0e1f2cd7cf96d3020} Started: Started container init-volume-dynamicpv-hbs7 Mar 27 23:50:03.283: INFO: At 2023-03-27 23:49:40 +0000 UTC - event for pod-subpath-test-dynamicpv-hbs7: {kubelet i-0e1f2cd7cf96d3020} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 27 23:50:03.283: INFO: At 2023-03-27 23:49:40 +0000 UTC - event for pod-subpath-test-dynamicpv-hbs7: {kubelet i-0e1f2cd7cf96d3020} Created: Created container test-init-volume-dynamicpv-hbs7 Mar 27 23:50:03.283: INFO: At 2023-03-27 23:49:41 +0000 UTC - event for pod-subpath-test-dynamicpv-hbs7: {kubelet i-0e1f2cd7cf96d3020} Started: Started container test-init-volume-dynamicpv-hbs7 Mar 27 23:50:03.283: INFO: At 2023-03-27 23:49:41 +0000 UTC - event for pod-subpath-test-dynamicpv-hbs7: {kubelet i-0e1f2cd7cf96d3020} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 27 23:50:03.283: INFO: At 2023-03-27 23:49:41 +0000 UTC - event for pod-subpath-test-dynamicpv-hbs7: {kubelet i-0e1f2cd7cf96d3020} Created: Created container test-container-subpath-dynamicpv-hbs7 Mar 27 23:50:03.283: INFO: At 2023-03-27 23:49:42 +0000 UTC - event for pod-subpath-test-dynamicpv-hbs7: {kubelet i-0e1f2cd7cf96d3020} Started: Started container test-container-subpath-dynamicpv-hbs7 Mar 27 23:50:03.459: INFO: POD NODE PHASE GRACE CONDITIONS Mar 27 23:50:03.459: INFO: Mar 27 23:50:03.637: INFO: Logging node info for node i-022003b3a45876d5d Mar 27 23:50:03.813: INFO: Node Info: &Node{ObjectMeta:{i-022003b3a45876d5d 20348b58-b610-41a0-accb-ca195a540336 43998 0 2023-03-27 23:33:26 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-022003b3a45876d5d kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-022003b3a45876d5d topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-022003b3a45876d5d"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.49.118/19 projectcalico.org/IPv4IPIPTunnelAddr:100.115.192.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:33:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:33:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:49:50 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:49:55 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-022003b3a45876d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:33:51 +0000 UTC,LastTransitionTime:2023-03-27 23:33:51 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:49:55 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:49:55 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:49:55 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:49:55 +0000 UTC,LastTransitionTime:2023-03-27 23:33:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.118,},NodeAddress{Type:ExternalIP,Address:54.180.103.89,},NodeAddress{Type:InternalDNS,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-103-89.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccb51411bbbb92946da93b9a452fe,SystemUUID:ec2ccb51-411b-bbb9-2946-da93b9a452fe,BootID:841d91a9-bfa5-4e0f-9279-7c83efeef8ca,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-09893b24a666ef116],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-09893b24a666ef116,DevicePath:,},},Config:nil,},} Mar 27 23:50:03.813: INFO: Logging kubelet events for node i-022003b3a45876d5d Mar 27 23:50:04.021: INFO: Logging pods the kubelet thinks is on node i-022003b3a45876d5d Mar 27 23:50:04.203: INFO: calico-node-2wjdb started at 2023-03-27 23:33:27 +0000 UTC (3+1 container statuses recorded) Mar 27 23:50:04.203: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:50:04.203: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:50:04.203: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:50:04.203: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:50:04.203: INFO: exec-volume-test-preprovisionedpv-8b4n started at 2023-03-27 23:50:01 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:04.203: INFO: Container exec-container-preprovisionedpv-8b4n ready: false, restart count 0 Mar 27 23:50:04.203: INFO: dns-test-72d27bff-f831-4477-ad2f-f7fbc3af3c78 started at 2023-03-27 23:49:55 +0000 UTC (0+3 container statuses recorded) Mar 27 23:50:04.203: INFO: Container jessie-querier ready: true, restart count 0 Mar 27 23:50:04.203: INFO: Container querier ready: true, restart count 0 Mar 27 23:50:04.203: INFO: Container webserver ready: true, restart count 0 Mar 27 23:50:04.203: INFO: kube-proxy-i-022003b3a45876d5d started at 2023-03-27 23:33:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:04.203: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:50:04.203: INFO: ebs-csi-node-2knd4 started at 2023-03-27 23:33:27 +0000 UTC (0+3 container statuses recorded) Mar 27 23:50:04.203: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:50:04.203: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:50:04.203: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:50:04.203: INFO: labelsupdate18826d69-79ed-419a-a047-07c13c56ec34 started at 2023-03-27 23:49:45 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:04.203: INFO: Container client-container ready: true, restart count 0 Mar 27 23:50:04.203: INFO: netserver-0 started at 2023-03-27 23:49:54 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:04.203: INFO: Container webserver ready: false, restart count 0 Mar 27 23:50:04.203: INFO: pvc-tester-4tjq5 started at 2023-03-27 23:50:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:04.203: INFO: Container write-pod ready: false, restart count 0 Mar 27 23:50:04.203: INFO: hostexec-i-022003b3a45876d5d-8ffm9 started at 2023-03-27 23:49:45 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:04.203: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:50:04.203: INFO: rs-tc5h9 started at 2023-03-27 23:49:54 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:04.203: INFO: Container donothing ready: true, restart count 0 Mar 27 23:50:04.826: INFO: Latency metrics for node i-022003b3a45876d5d Mar 27 23:50:04.826: INFO: Logging node info for node i-05ec0d9f3272223ec Mar 27 23:50:05.002: INFO: Node Info: &Node{ObjectMeta:{i-05ec0d9f3272223ec 365deda4-148e-4593-abdf-3103630fef28 43032 0 2023-03-27 23:25:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-05ec0d9f3272223ec kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-05ec0d9f3272223ec topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-capacity-3912":"csi-mock-csi-mock-volumes-capacity-3912","ebs.csi.aws.com":"i-05ec0d9f3272223ec"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.52.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.113.211.128 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:25:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:25:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:49:17 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:49:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-05ec0d9f3272223ec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:25:42 +0000 UTC,LastTransitionTime:2023-03-27 23:25:42 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:49:17 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:49:17 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:49:17 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:49:17 +0000 UTC,LastTransitionTime:2023-03-27 23:25:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.36,},NodeAddress{Type:ExternalIP,Address:3.34.125.198,},NodeAddress{Type:InternalDNS,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-34-125-198.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dea8f8b5e44e8fd6ef34a7ee9507f,SystemUUID:ec2dea8f-8b5e-44e8-fd6e-f34a7ee9507f,BootID:d41641dd-09ee-4029-b1ae-fb0f0a4ef62f,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d,DevicePath:,},},Config:nil,},} Mar 27 23:50:05.002: INFO: Logging kubelet events for node i-05ec0d9f3272223ec Mar 27 23:50:05.186: INFO: Logging pods the kubelet thinks is on node i-05ec0d9f3272223ec Mar 27 23:50:05.372: INFO: test-grpc-3f4e7a39-e8bd-4296-bd56-c61876e85beb started at 2023-03-27 23:48:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:05.372: INFO: Container agnhost ready: true, restart count 0 Mar 27 23:50:05.372: INFO: netserver-1 started at 2023-03-27 23:49:54 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:05.372: INFO: Container webserver ready: false, restart count 0 Mar 27 23:50:05.372: INFO: hostexec-i-05ec0d9f3272223ec-w6qsh started at 2023-03-27 23:49:56 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:05.372: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:50:05.372: INFO: kube-proxy-i-05ec0d9f3272223ec started at 2023-03-27 23:25:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:05.372: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:50:05.372: INFO: coredns-559769c974-m6hh6 started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:05.372: INFO: Container coredns ready: true, restart count 0 Mar 27 23:50:05.372: INFO: inline-volume-tester-kwwd2 started at 2023-03-27 23:43:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:05.372: INFO: Container csi-volume-tester ready: false, restart count 0 Mar 27 23:50:05.372: INFO: ebs-csi-node-gjw52 started at 2023-03-27 23:25:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:50:05.372: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:50:05.372: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:50:05.372: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:50:05.372: INFO: test-webserver-d61a8cc2-59e4-4516-aa86-d6b94f858278 started at 2023-03-27 23:47:23 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:05.372: INFO: Container test-webserver ready: true, restart count 0 Mar 27 23:50:05.372: INFO: csi-mockplugin-0 started at 2023-03-27 23:49:25 +0000 UTC (0+4 container statuses recorded) Mar 27 23:50:05.372: INFO: Container busybox ready: true, restart count 0 Mar 27 23:50:05.372: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:50:05.372: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:50:05.372: INFO: Container mock ready: true, restart count 0 Mar 27 23:50:05.372: INFO: pod-handle-http-request started at 2023-03-27 23:48:57 +0000 UTC (0+2 container statuses recorded) Mar 27 23:50:05.372: INFO: Container container-handle-http-request ready: true, restart count 0 Mar 27 23:50:05.372: INFO: Container container-handle-https-request ready: true, restart count 0 Mar 27 23:50:05.372: INFO: calico-node-5lq5m started at 2023-03-27 23:25:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:50:05.372: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:50:05.372: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:50:05.372: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:50:05.372: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:50:05.372: INFO: hostexec-i-05ec0d9f3272223ec-2hkzf started at 2023-03-27 23:50:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:05.372: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:50:05.372: INFO: pod-with-poststart-http-hook started at 2023-03-27 23:49:00 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:05.372: INFO: Container pod-with-poststart-http-hook ready: true, restart count 0 Mar 27 23:50:06.093: INFO: Latency metrics for node i-05ec0d9f3272223ec Mar 27 23:50:06.093: INFO: Logging node info for node i-0b266cca2f40af1c1 Mar 27 23:50:06.274: INFO: Node Info: &Node{ObjectMeta:{i-0b266cca2f40af1c1 5f957448-9afc-4966-b9d6-82332b7a159d 44254 0 2023-03-27 23:29:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0b266cca2f40af1c1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0b266cca2f40af1c1 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0b266cca2f40af1c1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.116.128.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:30:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:30:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:50:02 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:50:02 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0b266cca2f40af1c1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:30:13 +0000 UTC,LastTransitionTime:2023-03-27 23:30:13 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:02 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:02 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:02 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:50:02 +0000 UTC,LastTransitionTime:2023-03-27 23:30:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.36,},NodeAddress{Type:ExternalIP,Address:13.209.22.95,},NodeAddress{Type:InternalDNS,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-209-22-95.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec247601b7b315369bfb310559a7772d,SystemUUID:ec247601-b7b3-1536-9bfb-310559a7772d,BootID:487b59c3-9394-4a90-a6ec-b20c4819fe9e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0d796a18722605b02],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0d796a18722605b02,DevicePath:,},},Config:nil,},} Mar 27 23:50:06.274: INFO: Logging kubelet events for node i-0b266cca2f40af1c1 Mar 27 23:50:06.465: INFO: Logging pods the kubelet thinks is on node i-0b266cca2f40af1c1 Mar 27 23:50:06.647: INFO: kube-proxy-i-0b266cca2f40af1c1 started at 2023-03-27 23:29:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:06.647: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:50:06.647: INFO: calico-node-nmfrn started at 2023-03-27 23:29:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:50:06.647: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:50:06.647: INFO: Init container install-cni ready: true, restart count 1 Mar 27 23:50:06.647: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:50:06.647: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:50:06.647: INFO: pod-configmaps-469afe86-b23e-4133-b521-f7d9cffdedcf started at 2023-03-27 23:46:30 +0000 UTC (0+2 container statuses recorded) Mar 27 23:50:06.647: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:50:06.647: INFO: Container configmap-volume-binary-test ready: false, restart count 0 Mar 27 23:50:06.647: INFO: pod-9e186fe4-4c36-475e-91aa-531497882d26 started at 2023-03-27 23:49:56 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:06.647: INFO: Container write-pod ready: true, restart count 0 Mar 27 23:50:06.647: INFO: pod-subpath-test-inlinevolume-qkpt started at 2023-03-27 23:50:06 +0000 UTC (1+1 container statuses recorded) Mar 27 23:50:06.647: INFO: Init container init-volume-inlinevolume-qkpt ready: false, restart count 0 Mar 27 23:50:06.647: INFO: Container test-container-subpath-inlinevolume-qkpt ready: false, restart count 0 Mar 27 23:50:06.647: INFO: rs-v6prb started at 2023-03-27 23:49:54 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:06.647: INFO: Container donothing ready: true, restart count 0 Mar 27 23:50:06.647: INFO: ebs-csi-node-ztjm9 started at 2023-03-27 23:29:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:50:06.647: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:50:06.647: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:50:06.647: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:50:06.647: INFO: busybox-readonly-fs95a19e50-5cef-4e59-8e66-aa79c28512cd started at 2023-03-27 23:49:55 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:06.647: INFO: Container busybox-readonly-fs95a19e50-5cef-4e59-8e66-aa79c28512cd ready: true, restart count 0 Mar 27 23:50:06.647: INFO: netserver-2 started at 2023-03-27 23:49:54 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:06.647: INFO: Container webserver ready: false, restart count 0 Mar 27 23:50:06.647: INFO: hostexec-i-0b266cca2f40af1c1-b8bsc started at 2023-03-27 23:49:56 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:06.647: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:50:07.325: INFO: Latency metrics for node i-0b266cca2f40af1c1 Mar 27 23:50:07.325: INFO: Logging node info for node i-0d2b1184f62de758a Mar 27 23:50:07.502: INFO: Node Info: &Node{ObjectMeta:{i-0d2b1184f62de758a 2e42ccb1-9580-4126-b5d6-05a5f98403c5 31490 0 2023-03-27 23:17:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d2b1184f62de758a kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d2b1184f62de758a"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.6/19 projectcalico.org/IPv4IPIPTunnelAddr:100.120.150.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-27 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-27 23:17:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:18:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:45:46 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0d2b1184f62de758a,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:18:49 +0000 UTC,LastTransitionTime:2023-03-27 23:18:49 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:45:46 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:45:46 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:45:46 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:45:46 +0000 UTC,LastTransitionTime:2023-03-27 23:18:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.6,},NodeAddress{Type:ExternalIP,Address:3.35.11.126,},NodeAddress{Type:InternalDNS,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-35-11-126.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2781baf4898d15db3c853bf5b6d100,SystemUUID:ec2781ba-f489-8d15-db3c-853bf5b6d100,BootID:5e905df0-494e-4050-b6d7-400763c3dd66,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45208814,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43629012,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074727,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:50:07.502: INFO: Logging kubelet events for node i-0d2b1184f62de758a Mar 27 23:50:07.705: INFO: Logging pods the kubelet thinks is on node i-0d2b1184f62de758a Mar 27 23:50:07.889: INFO: kops-controller-w4b7t started at 2023-03-27 23:18:29 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:07.889: INFO: Container kops-controller ready: true, restart count 0 Mar 27 23:50:07.889: INFO: kube-scheduler-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:07.889: INFO: Container kube-scheduler ready: true, restart count 0 Mar 27 23:50:07.889: INFO: dns-controller-69987775c6-rzzjj started at 2023-03-27 23:18:25 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:07.889: INFO: Container dns-controller ready: true, restart count 0 Mar 27 23:50:07.889: INFO: calico-node-scbv2 started at 2023-03-27 23:18:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:50:07.889: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:50:07.889: INFO: ebs-csi-node-nvqcx started at 2023-03-27 23:18:29 +0000 UTC (0+3 container statuses recorded) Mar 27 23:50:07.889: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:50:07.889: INFO: kube-proxy-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:07.889: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:50:07.889: INFO: aws-cloud-controller-manager-776ld started at 2023-03-27 23:18:28 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:07.889: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 27 23:50:07.889: INFO: calico-kube-controllers-55f488656-vj6gh started at 2023-03-27 23:18:57 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:07.889: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 27 23:50:07.889: INFO: ebs-csi-controller-79f984545f-42vd6 started at 2023-03-27 23:19:48 +0000 UTC (0+5 container statuses recorded) Mar 27 23:50:07.889: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:50:07.889: INFO: etcd-manager-events-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:50:07.889: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:50:07.889: INFO: etcd-manager-main-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:50:07.889: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:50:07.889: INFO: kube-apiserver-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+2 container statuses recorded) Mar 27 23:50:07.889: INFO: Container healthcheck ready: true, restart count 0 Mar 27 23:50:07.889: INFO: Container kube-apiserver ready: true, restart count 3 Mar 27 23:50:07.889: INFO: kube-controller-manager-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:07.889: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 27 23:50:08.494: INFO: Latency metrics for node i-0d2b1184f62de758a Mar 27 23:50:08.494: INFO: Logging node info for node i-0e1f2cd7cf96d3020 Mar 27 23:50:08.671: INFO: Node Info: &Node{ObjectMeta:{i-0e1f2cd7cf96d3020 efcf1e66-db0e-4256-a849-b9c377eef04e 43744 0 2023-03-27 23:22:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e1f2cd7cf96d3020 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0e1f2cd7cf96d3020 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-expansion-4700":"i-0e1f2cd7cf96d3020","ebs.csi.aws.com":"i-0e1f2cd7cf96d3020"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.120/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.86.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:22:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:22:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:49:49 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:49:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0e1f2cd7cf96d3020,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:22:52 +0000 UTC,LastTransitionTime:2023-03-27 23:22:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:49:49 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:49:49 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:49:49 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:49:49 +0000 UTC,LastTransitionTime:2023-03-27 23:22:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.120,},NodeAddress{Type:ExternalIP,Address:15.164.224.157,},NodeAddress{Type:InternalDNS,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-164-224-157.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29484c87c97fb3d8a52788d5971660,SystemUUID:ec29484c-87c9-7fb3-d8a5-2788d5971660,BootID:b73c540f-bd49-470a-a7b3-08c31cc8c2fd,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4700^301e3bbe-ccf9-11ed-a882-f233d05cf674],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-4700^301e3bbe-ccf9-11ed-a882-f233d05cf674,DevicePath:,},},Config:nil,},} Mar 27 23:50:08.671: INFO: Logging kubelet events for node i-0e1f2cd7cf96d3020 Mar 27 23:50:08.857: INFO: Logging pods the kubelet thinks is on node i-0e1f2cd7cf96d3020 Mar 27 23:50:09.039: INFO: csi-mockplugin-resizer-0 started at 2023-03-27 23:43:31 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:09.039: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:50:09.039: INFO: coredns-autoscaler-7cb5c5b969-2k8xf started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:09.039: INFO: Container autoscaler ready: true, restart count 0 Mar 27 23:50:09.039: INFO: pod-07cb0dee-dc03-40ae-a2be-dd6cb75d52b4 started at 2023-03-27 23:50:05 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:09.039: INFO: Container write-pod ready: true, restart count 0 Mar 27 23:50:09.039: INFO: csi-mockplugin-attacher-0 started at 2023-03-27 23:43:31 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:09.039: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:50:09.039: INFO: netserver-3 started at 2023-03-27 23:49:54 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:09.039: INFO: Container webserver ready: true, restart count 0 Mar 27 23:50:09.039: INFO: csi-mockplugin-0 started at 2023-03-27 23:43:31 +0000 UTC (0+3 container statuses recorded) Mar 27 23:50:09.039: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:50:09.039: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:50:09.039: INFO: Container mock ready: true, restart count 0 Mar 27 23:50:09.039: INFO: rs-vg4vd started at 2023-03-27 23:50:00 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:09.039: INFO: Container donothing ready: true, restart count 0 Mar 27 23:50:09.039: INFO: pvc-volume-tester-cwxsw started at 2023-03-27 23:43:35 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:09.039: INFO: Container volume-tester ready: true, restart count 0 Mar 27 23:50:09.039: INFO: coredns-559769c974-scn7m started at 2023-03-27 23:30:58 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:09.039: INFO: Container coredns ready: true, restart count 0 Mar 27 23:50:09.039: INFO: hostexec-i-0e1f2cd7cf96d3020-756nf started at 2023-03-27 23:50:00 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:09.039: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:50:09.039: INFO: ebs-csi-node-vcn4h started at 2023-03-27 23:22:28 +0000 UTC (0+3 container statuses recorded) Mar 27 23:50:09.039: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:50:09.039: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:50:09.039: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:50:09.039: INFO: calico-node-9fzr7 started at 2023-03-27 23:22:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:50:09.039: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:50:09.039: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:50:09.039: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:50:09.039: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:50:09.039: INFO: kube-proxy-i-0e1f2cd7cf96d3020 started at 2023-03-27 23:22:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:50:09.039: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:50:09.691: INFO: Latency metrics for node i-0e1f2cd7cf96d3020 END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:50:09.691 (6.584s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/27/23 23:50:09.691 (6.584s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/27/23 23:50:09.691 STEP: Destroying namespace "provisioning-1057" for this suite. - test/e2e/framework/framework.go:351 @ 03/27/23 23:50:09.691 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/27/23 23:50:09.871 (180ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:50:09.871 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:50:09.871 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\ssubPath\sshould\ssupport\sreadOnly\sfile\sspecified\sin\sthe\svolumeMount\s\[LinuxOnly\]$'
[FAILED] failed to get logs from pod-subpath-test-dynamicpv-s6fg for test-container-subpath-dynamicpv-s6fg: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-2337/pods/pod-subpath-test-dynamicpv-s6fg/log?container=test-container-subpath-dynamicpv-s6fg&previous=false": stream error: stream ID 409; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:53:59.989from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:53:44.058 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:53:44.058 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/27/23 23:53:44.058 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/27/23 23:53:44.058 Mar 27 23:53:44.058: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename provisioning - test/e2e/framework/framework.go:250 @ 03/27/23 23:53:44.059 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/27/23 23:53:44.593 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/27/23 23:53:44.947 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - set up framework | framework.go:191 @ 03/27/23 23:53:45.3 (1.242s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:53:45.3 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:53:45.3 (0s) > Enter [It] should support readOnly file specified in the volumeMount [LinuxOnly] - test/e2e/storage/testsuites/subpath.go:382 @ 03/27/23 23:53:45.3 Mar 27 23:53:45.300: INFO: Creating resource for dynamic PV Mar 27 23:53:45.300: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass provisioning-2337-e2e-scl22lv - test/e2e/storage/framework/volume_resource.go:102 @ 03/27/23 23:53:45.301 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/27/23 23:53:45.482 Mar 27 23:53:45.482: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod pod-subpath-test-dynamicpv-s6fg - test/e2e/storage/testsuites/subpath.go:691 @ 03/27/23 23:53:45.839 STEP: Creating a pod to test subpath - test/e2e/framework/pod/output/output.go:243 @ 03/27/23 23:53:45.839 STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/27/23 23:53:59.27 Mar 27 23:53:59.449: INFO: Trying to get logs from node i-05ec0d9f3272223ec pod pod-subpath-test-dynamicpv-s6fg container test-container-subpath-dynamicpv-s6fg: <nil> Mar 27 23:53:59.626: INFO: Failed to get logs from node "i-05ec0d9f3272223ec" pod "pod-subpath-test-dynamicpv-s6fg" container "test-container-subpath-dynamicpv-s6fg". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-2337/pods/pod-subpath-test-dynamicpv-s6fg/log?container=test-container-subpath-dynamicpv-s6fg&previous=false": stream error: stream ID 409; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/27/23 23:53:59.626 Mar 27 23:53:59.989: INFO: Unexpected error: <*fmt.wrapError | 0xc007900440>: failed to get logs from pod-subpath-test-dynamicpv-s6fg for test-container-subpath-dynamicpv-s6fg: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-2337/pods/pod-subpath-test-dynamicpv-s6fg/log?container=test-container-subpath-dynamicpv-s6fg&previous=false": stream error: stream ID 409; INTERNAL_ERROR; received from peer { msg: "failed to get logs from pod-subpath-test-dynamicpv-s6fg for test-container-subpath-dynamicpv-s6fg: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-2337/pods/pod-subpath-test-dynamicpv-s6fg/log?container=test-container-subpath-dynamicpv-s6fg&previous=false\": stream error: stream ID 409; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc0022d8d20>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-2337/pods/pod-subpath-test-dynamicpv-s6fg/log?container=test-container-subpath-dynamicpv-s6fg&previous=false", Err: <http2.StreamError>{ StreamID: 409, Code: 2, Cause: <*errors.errorString | 0xc0001e7680>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from pod-subpath-test-dynamicpv-s6fg for test-container-subpath-dynamicpv-s6fg: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/provisioning-2337/pods/pod-subpath-test-dynamicpv-s6fg/log?container=test-container-subpath-dynamicpv-s6fg&previous=false": stream error: stream ID 409; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:53:59.989 < Exit [It] should support readOnly file specified in the volumeMount [LinuxOnly] - test/e2e/storage/testsuites/subpath.go:382 @ 03/27/23 23:53:59.989 (14.689s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:53:59.989 Mar 27 23:53:59.989: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:54:00.167 (178ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:384 @ 03/27/23 23:54:00.168 STEP: Deleting pod - test/e2e/storage/testsuites/subpath.go:172 @ 03/27/23 23:54:00.168 Mar 27 23:54:00.168: INFO: Deleting pod "pod-subpath-test-dynamicpv-s6fg" in namespace "provisioning-2337" STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/27/23 23:54:00.52 Mar 27 23:54:00.887: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.com67fkl" Mar 27 23:54:01.067: INFO: Waiting up to 5m0s for PersistentVolume pvc-588e65ca-b861-4aae-a905-4675831b62ff to get deleted Mar 27 23:54:01.246: INFO: PersistentVolume pvc-588e65ca-b861-4aae-a905-4675831b62ff found and phase=Released (179.0543ms) Mar 27 23:54:06.427: INFO: PersistentVolume pvc-588e65ca-b861-4aae-a905-4675831b62ff found and phase=Released (5.359956841s) Mar 27 23:54:11.605: INFO: PersistentVolume pvc-588e65ca-b861-4aae-a905-4675831b62ff found and phase=Released (10.537607791s) Mar 27 23:54:16.782: INFO: PersistentVolume pvc-588e65ca-b861-4aae-a905-4675831b62ff was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/27/23 23:54:16.782 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/storage/testsuites/subpath.go:384 @ 03/27/23 23:54:16.96 (16.793s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:54:16.96 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:54:16.96 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/27/23 23:54:16.96 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:54:16.961 STEP: Collecting events from namespace "provisioning-2337". - test/e2e/framework/debug/dump.go:42 @ 03/27/23 23:54:16.961 STEP: Found 15 events. - test/e2e/framework/debug/dump.go:46 @ 03/27/23 23:54:17.138 Mar 27 23:54:17.138: INFO: At 2023-03-27 23:53:45 +0000 UTC - event for ebs.csi.aws.com67fkl: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 27 23:54:17.138: INFO: At 2023-03-27 23:53:45 +0000 UTC - event for ebs.csi.aws.com67fkl: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 27 23:54:17.138: INFO: At 2023-03-27 23:53:45 +0000 UTC - event for ebs.csi.aws.com67fkl: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } Provisioning: External provisioner is provisioning volume for claim "provisioning-2337/ebs.csi.aws.com67fkl" Mar 27 23:54:17.138: INFO: At 2023-03-27 23:53:49 +0000 UTC - event for ebs.csi.aws.com67fkl: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } ProvisioningSucceeded: Successfully provisioned volume pvc-588e65ca-b861-4aae-a905-4675831b62ff Mar 27 23:54:17.138: INFO: At 2023-03-27 23:53:49 +0000 UTC - event for pod-subpath-test-dynamicpv-s6fg: {default-scheduler } Scheduled: Successfully assigned provisioning-2337/pod-subpath-test-dynamicpv-s6fg to i-05ec0d9f3272223ec Mar 27 23:54:17.138: INFO: At 2023-03-27 23:53:52 +0000 UTC - event for pod-subpath-test-dynamicpv-s6fg: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-588e65ca-b861-4aae-a905-4675831b62ff" Mar 27 23:54:17.138: INFO: At 2023-03-27 23:53:54 +0000 UTC - event for pod-subpath-test-dynamicpv-s6fg: {kubelet i-05ec0d9f3272223ec} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 27 23:54:17.138: INFO: At 2023-03-27 23:53:54 +0000 UTC - event for pod-subpath-test-dynamicpv-s6fg: {kubelet i-05ec0d9f3272223ec} Created: Created container init-volume-dynamicpv-s6fg Mar 27 23:54:17.138: INFO: At 2023-03-27 23:53:54 +0000 UTC - event for pod-subpath-test-dynamicpv-s6fg: {kubelet i-05ec0d9f3272223ec} Started: Started container init-volume-dynamicpv-s6fg Mar 27 23:54:17.138: INFO: At 2023-03-27 23:53:54 +0000 UTC - event for pod-subpath-test-dynamicpv-s6fg: {kubelet i-05ec0d9f3272223ec} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 27 23:54:17.138: INFO: At 2023-03-27 23:53:54 +0000 UTC - event for pod-subpath-test-dynamicpv-s6fg: {kubelet i-05ec0d9f3272223ec} Created: Created container test-init-volume-dynamicpv-s6fg Mar 27 23:54:17.138: INFO: At 2023-03-27 23:53:54 +0000 UTC - event for pod-subpath-test-dynamicpv-s6fg: {kubelet i-05ec0d9f3272223ec} Started: Started container test-init-volume-dynamicpv-s6fg Mar 27 23:54:17.138: INFO: At 2023-03-27 23:53:55 +0000 UTC - event for pod-subpath-test-dynamicpv-s6fg: {kubelet i-05ec0d9f3272223ec} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 27 23:54:17.138: INFO: At 2023-03-27 23:53:55 +0000 UTC - event for pod-subpath-test-dynamicpv-s6fg: {kubelet i-05ec0d9f3272223ec} Created: Created container test-container-subpath-dynamicpv-s6fg Mar 27 23:54:17.138: INFO: At 2023-03-27 23:53:56 +0000 UTC - event for pod-subpath-test-dynamicpv-s6fg: {kubelet i-05ec0d9f3272223ec} Started: Started container test-container-subpath-dynamicpv-s6fg Mar 27 23:54:17.315: INFO: POD NODE PHASE GRACE CONDITIONS Mar 27 23:54:17.315: INFO: Mar 27 23:54:17.518: INFO: Logging node info for node i-022003b3a45876d5d Mar 27 23:54:17.710: INFO: Node Info: &Node{ObjectMeta:{i-022003b3a45876d5d 20348b58-b610-41a0-accb-ca195a540336 54569 0 2023-03-27 23:33:26 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-022003b3a45876d5d kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-022003b3a45876d5d topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-022003b3a45876d5d"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.49.118/19 projectcalico.org/IPv4IPIPTunnelAddr:100.115.192.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:33:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:33:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:53:49 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:53:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-022003b3a45876d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:33:51 +0000 UTC,LastTransitionTime:2023-03-27 23:33:51 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:49 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:49 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:49 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:53:49 +0000 UTC,LastTransitionTime:2023-03-27 23:33:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.118,},NodeAddress{Type:ExternalIP,Address:54.180.103.89,},NodeAddress{Type:InternalDNS,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-103-89.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccb51411bbbb92946da93b9a452fe,SystemUUID:ec2ccb51-411b-bbb9-2946-da93b9a452fe,BootID:841d91a9-bfa5-4e0f-9279-7c83efeef8ca,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-032846faeb20c668e],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-032846faeb20c668e,DevicePath:,},},Config:nil,},} Mar 27 23:54:17.711: INFO: Logging kubelet events for node i-022003b3a45876d5d Mar 27 23:54:17.905: INFO: Logging pods the kubelet thinks is on node i-022003b3a45876d5d Mar 27 23:54:18.100: INFO: kube-proxy-i-022003b3a45876d5d started at 2023-03-27 23:33:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:18.100: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:54:18.100: INFO: ebs-csi-node-2knd4 started at 2023-03-27 23:33:27 +0000 UTC (0+3 container statuses recorded) Mar 27 23:54:18.100: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:54:18.100: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:54:18.100: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:54:18.100: INFO: all-succeed-xj5gq started at 2023-03-27 23:54:05 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:18.100: INFO: Container c ready: false, restart count 0 Mar 27 23:54:18.100: INFO: pfpod started at 2023-03-27 23:52:53 +0000 UTC (0+2 container statuses recorded) Mar 27 23:54:18.100: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:54:18.100: INFO: Container readiness ready: false, restart count 0 Mar 27 23:54:18.100: INFO: netserver-0 started at 2023-03-27 23:54:07 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:18.100: INFO: Container webserver ready: false, restart count 0 Mar 27 23:54:18.100: INFO: pfpod started at 2023-03-27 23:53:13 +0000 UTC (0+2 container statuses recorded) Mar 27 23:54:18.100: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:54:18.100: INFO: Container readiness ready: false, restart count 0 Mar 27 23:54:18.100: INFO: pod-3c7b65ab-744f-4a60-b98d-d02d93a90be0 started at 2023-03-27 23:52:51 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:18.100: INFO: Container write-pod ready: true, restart count 0 Mar 27 23:54:18.100: INFO: all-succeed-rlvnq started at 2023-03-27 23:54:09 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:18.100: INFO: Container c ready: false, restart count 0 Mar 27 23:54:18.100: INFO: calico-node-2wjdb started at 2023-03-27 23:33:27 +0000 UTC (3+1 container statuses recorded) Mar 27 23:54:18.100: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:54:18.100: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:54:18.100: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:54:18.100: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:54:18.100: INFO: webserver-pod started at 2023-03-27 23:52:55 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:18.100: INFO: Container agnhost ready: false, restart count 0 Mar 27 23:54:18.100: INFO: inline-volume-p4k8g started at 2023-03-27 23:52:15 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:18.100: INFO: Container volume-tester ready: false, restart count 0 Mar 27 23:54:18.100: INFO: up-down-1-sqrz2 started at 2023-03-27 23:54:15 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:18.100: INFO: Container up-down-1 ready: true, restart count 0 Mar 27 23:54:18.726: INFO: Latency metrics for node i-022003b3a45876d5d Mar 27 23:54:18.726: INFO: Logging node info for node i-05ec0d9f3272223ec Mar 27 23:54:18.903: INFO: Node Info: &Node{ObjectMeta:{i-05ec0d9f3272223ec 365deda4-148e-4593-abdf-3103630fef28 55208 0 2023-03-27 23:25:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-05ec0d9f3272223ec kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-05ec0d9f3272223ec topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-05ec0d9f3272223ec"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.52.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.113.211.128 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:25:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:25:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:54:03 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:54:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-05ec0d9f3272223ec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:25:42 +0000 UTC,LastTransitionTime:2023-03-27 23:25:42 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:03 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:03 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:03 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:54:03 +0000 UTC,LastTransitionTime:2023-03-27 23:25:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.36,},NodeAddress{Type:ExternalIP,Address:3.34.125.198,},NodeAddress{Type:InternalDNS,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-34-125-198.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dea8f8b5e44e8fd6ef34a7ee9507f,SystemUUID:ec2dea8f-8b5e-44e8-fd6e-f34a7ee9507f,BootID:d41641dd-09ee-4029-b1ae-fb0f0a4ef62f,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d,DevicePath:,},},Config:nil,},} Mar 27 23:54:18.904: INFO: Logging kubelet events for node i-05ec0d9f3272223ec Mar 27 23:54:19.100: INFO: Logging pods the kubelet thinks is on node i-05ec0d9f3272223ec Mar 27 23:54:19.285: INFO: labelsupdate7e586ed3-d6b2-4b24-9a28-0256a577e654 started at 2023-03-27 23:52:19 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:19.285: INFO: Container client-container ready: true, restart count 0 Mar 27 23:54:19.285: INFO: pod1 started at 2023-03-27 23:53:57 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:19.285: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:54:19.285: INFO: all-succeed-csvk5 started at 2023-03-27 23:54:05 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:19.285: INFO: Container c ready: false, restart count 0 Mar 27 23:54:19.285: INFO: calico-node-5lq5m started at 2023-03-27 23:25:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:54:19.285: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:54:19.285: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:54:19.285: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:54:19.285: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:54:19.285: INFO: execpodnrx8g started at 2023-03-27 23:54:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:19.285: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:54:19.285: INFO: pod-subpath-test-dynamicpv-ngj9 started at 2023-03-27 23:54:16 +0000 UTC (1+1 container statuses recorded) Mar 27 23:54:19.285: INFO: Init container init-volume-dynamicpv-ngj9 ready: false, restart count 0 Mar 27 23:54:19.285: INFO: Container test-container-subpath-dynamicpv-ngj9 ready: false, restart count 0 Mar 27 23:54:19.285: INFO: netserver-1 started at 2023-03-27 23:54:07 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:19.285: INFO: Container webserver ready: true, restart count 0 Mar 27 23:54:19.285: INFO: all-succeed-nrjf6 started at 2023-03-27 23:54:09 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:19.285: INFO: Container c ready: false, restart count 0 Mar 27 23:54:19.285: INFO: termination-message-container4e59df06-cea4-4c64-8174-b085ad49f08c started at 2023-03-27 23:54:15 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:19.285: INFO: Container termination-message-container ready: false, restart count 0 Mar 27 23:54:19.285: INFO: up-down-1-6p6rl started at 2023-03-27 23:54:15 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:19.285: INFO: Container up-down-1 ready: true, restart count 0 Mar 27 23:54:19.285: INFO: kube-proxy-i-05ec0d9f3272223ec started at 2023-03-27 23:25:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:19.285: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:54:19.285: INFO: ebs-csi-node-gjw52 started at 2023-03-27 23:25:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:54:19.285: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:54:19.285: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:54:19.285: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:54:19.285: INFO: coredns-559769c974-m6hh6 started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:19.285: INFO: Container coredns ready: true, restart count 0 Mar 27 23:54:19.285: INFO: inline-volume-tester-kwwd2 started at 2023-03-27 23:43:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:19.285: INFO: Container csi-volume-tester ready: false, restart count 0 Mar 27 23:54:19.285: INFO: pod-ephm-test-secret-jtp6 started at 2023-03-27 23:53:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:19.285: INFO: Container test-container-subpath-secret-jtp6 ready: false, restart count 0 Mar 27 23:54:19.954: INFO: Latency metrics for node i-05ec0d9f3272223ec Mar 27 23:54:19.954: INFO: Logging node info for node i-0b266cca2f40af1c1 Mar 27 23:54:20.135: INFO: Node Info: &Node{ObjectMeta:{i-0b266cca2f40af1c1 5f957448-9afc-4966-b9d6-82332b7a159d 54652 0 2023-03-27 23:29:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0b266cca2f40af1c1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0b266cca2f40af1c1 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-expansion-2991":"i-0b266cca2f40af1c1","ebs.csi.aws.com":"i-0b266cca2f40af1c1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.116.128.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:30:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:30:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:53:44 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:53:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0b266cca2f40af1c1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:30:13 +0000 UTC,LastTransitionTime:2023-03-27 23:30:13 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:47 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:47 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:53:47 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:53:47 +0000 UTC,LastTransitionTime:2023-03-27 23:30:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.36,},NodeAddress{Type:ExternalIP,Address:13.209.22.95,},NodeAddress{Type:InternalDNS,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-209-22-95.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec247601b7b315369bfb310559a7772d,SystemUUID:ec247601-b7b3-1536-9bfb-310559a7772d,BootID:487b59c3-9394-4a90-a6ec-b20c4819fe9e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-2991^9ab3bf37-ccfa-11ed-a28d-ead8e583275f kubernetes.io/csi/ebs.csi.aws.com^vol-02340833e64aefd4f],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-2991^9ab3bf37-ccfa-11ed-a28d-ead8e583275f,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02340833e64aefd4f,DevicePath:,},},Config:nil,},} Mar 27 23:54:20.135: INFO: Logging kubelet events for node i-0b266cca2f40af1c1 Mar 27 23:54:20.321: INFO: Logging pods the kubelet thinks is on node i-0b266cca2f40af1c1 Mar 27 23:54:20.506: INFO: pod-projected-configmaps-13a5b767-9913-4450-bbfa-e347f6f38991 started at 2023-03-27 23:50:31 +0000 UTC (0+3 container statuses recorded) Mar 27 23:54:20.506: INFO: Container createcm-volume-test ready: true, restart count 0 Mar 27 23:54:20.506: INFO: Container delcm-volume-test ready: true, restart count 0 Mar 27 23:54:20.506: INFO: Container updcm-volume-test ready: true, restart count 0 Mar 27 23:54:20.506: INFO: inline-volume-tester-jh9w9 started at 2023-03-27 23:52:51 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:20.506: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 27 23:54:20.506: INFO: pfpod started at 2023-03-27 23:52:51 +0000 UTC (0+2 container statuses recorded) Mar 27 23:54:20.506: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:54:20.506: INFO: Container readiness ready: false, restart count 0 Mar 27 23:54:20.506: INFO: up-down-2-w7vfg started at 2023-03-27 23:54:19 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:20.506: INFO: Container up-down-2 ready: false, restart count 0 Mar 27 23:54:20.506: INFO: hostexec-i-0b266cca2f40af1c1-c4jl6 started at 2023-03-27 23:54:20 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:20.506: INFO: Container agnhost-container ready: false, restart count 0 Mar 27 23:54:20.506: INFO: calico-node-nmfrn started at 2023-03-27 23:29:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:54:20.506: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:54:20.506: INFO: Init container install-cni ready: true, restart count 1 Mar 27 23:54:20.506: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:54:20.506: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:54:20.506: INFO: kube-proxy-i-0b266cca2f40af1c1 started at 2023-03-27 23:29:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:20.506: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:54:20.506: INFO: csi-mockplugin-resizer-0 started at 2023-03-27 23:53:40 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:20.506: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:54:20.506: INFO: csi-mockplugin-0 started at 2023-03-27 23:53:39 +0000 UTC (0+3 container statuses recorded) Mar 27 23:54:20.506: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:54:20.506: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:54:20.506: INFO: Container mock ready: true, restart count 0 Mar 27 23:54:20.506: INFO: pvc-volume-tester-kjswl started at 2023-03-27 23:53:43 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:20.506: INFO: Container volume-tester ready: true, restart count 0 Mar 27 23:54:20.506: INFO: csi-mockplugin-attacher-0 started at 2023-03-27 23:53:39 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:20.506: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:54:20.506: INFO: netserver-2 started at 2023-03-27 23:54:07 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:20.506: INFO: Container webserver ready: true, restart count 0 Mar 27 23:54:20.506: INFO: ebs-csi-node-ztjm9 started at 2023-03-27 23:29:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:54:20.506: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:54:20.506: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:54:20.506: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:54:21.161: INFO: Latency metrics for node i-0b266cca2f40af1c1 Mar 27 23:54:21.161: INFO: Logging node info for node i-0d2b1184f62de758a Mar 27 23:54:21.339: INFO: Node Info: &Node{ObjectMeta:{i-0d2b1184f62de758a 2e42ccb1-9580-4126-b5d6-05a5f98403c5 45677 0 2023-03-27 23:17:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d2b1184f62de758a kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d2b1184f62de758a"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.6/19 projectcalico.org/IPv4IPIPTunnelAddr:100.120.150.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-27 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-27 23:17:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:18:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:50:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0d2b1184f62de758a,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:18:49 +0000 UTC,LastTransitionTime:2023-03-27 23:18:49 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:18:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.6,},NodeAddress{Type:ExternalIP,Address:3.35.11.126,},NodeAddress{Type:InternalDNS,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-35-11-126.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2781baf4898d15db3c853bf5b6d100,SystemUUID:ec2781ba-f489-8d15-db3c-853bf5b6d100,BootID:5e905df0-494e-4050-b6d7-400763c3dd66,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45208814,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43629012,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074727,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:54:21.340: INFO: Logging kubelet events for node i-0d2b1184f62de758a Mar 27 23:54:21.530: INFO: Logging pods the kubelet thinks is on node i-0d2b1184f62de758a Mar 27 23:54:21.714: INFO: kube-scheduler-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:21.714: INFO: Container kube-scheduler ready: true, restart count 0 Mar 27 23:54:21.714: INFO: dns-controller-69987775c6-rzzjj started at 2023-03-27 23:18:25 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:21.714: INFO: Container dns-controller ready: true, restart count 0 Mar 27 23:54:21.714: INFO: calico-node-scbv2 started at 2023-03-27 23:18:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:54:21.714: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:54:21.714: INFO: ebs-csi-node-nvqcx started at 2023-03-27 23:18:29 +0000 UTC (0+3 container statuses recorded) Mar 27 23:54:21.714: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:54:21.714: INFO: kops-controller-w4b7t started at 2023-03-27 23:18:29 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:21.714: INFO: Container kops-controller ready: true, restart count 0 Mar 27 23:54:21.714: INFO: etcd-manager-events-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:54:21.714: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:54:21.714: INFO: etcd-manager-main-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:54:21.714: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:54:21.714: INFO: kube-apiserver-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+2 container statuses recorded) Mar 27 23:54:21.714: INFO: Container healthcheck ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Container kube-apiserver ready: true, restart count 3 Mar 27 23:54:21.714: INFO: kube-controller-manager-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:21.714: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 27 23:54:21.714: INFO: kube-proxy-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:21.714: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:54:21.714: INFO: aws-cloud-controller-manager-776ld started at 2023-03-27 23:18:28 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:21.714: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 27 23:54:21.714: INFO: calico-kube-controllers-55f488656-vj6gh started at 2023-03-27 23:18:57 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:21.714: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 27 23:54:21.714: INFO: ebs-csi-controller-79f984545f-42vd6 started at 2023-03-27 23:19:48 +0000 UTC (0+5 container statuses recorded) Mar 27 23:54:21.714: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:54:21.714: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:54:22.339: INFO: Latency metrics for node i-0d2b1184f62de758a Mar 27 23:54:22.339: INFO: Logging node info for node i-0e1f2cd7cf96d3020 Mar 27 23:54:22.517: INFO: Node Info: &Node{ObjectMeta:{i-0e1f2cd7cf96d3020 efcf1e66-db0e-4256-a849-b9c377eef04e 55887 0 2023-03-27 23:22:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e1f2cd7cf96d3020 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0e1f2cd7cf96d3020 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-35":"i-0e1f2cd7cf96d3020","csi-mock-csi-mock-volumes-workload-711":"i-0e1f2cd7cf96d3020","ebs.csi.aws.com":"i-0e1f2cd7cf96d3020"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.120/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.86.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:22:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:22:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:54:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {kube-controller-manager Update v1 2023-03-27 23:54:20 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0e1f2cd7cf96d3020,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:22:52 +0000 UTC,LastTransitionTime:2023-03-27 23:22:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:05 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:05 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:05 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:54:05 +0000 UTC,LastTransitionTime:2023-03-27 23:22:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.120,},NodeAddress{Type:ExternalIP,Address:15.164.224.157,},NodeAddress{Type:InternalDNS,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-164-224-157.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29484c87c97fb3d8a52788d5971660,SystemUUID:ec29484c-87c9-7fb3-d8a5-2788d5971660,BootID:b73c540f-bd49-470a-a7b3-08c31cc8c2fd,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-provisioning-35^b0054917-ccfa-11ed-9853-b67719a93dcf,DevicePath:,},},Config:nil,},} Mar 27 23:54:22.517: INFO: Logging kubelet events for node i-0e1f2cd7cf96d3020 Mar 27 23:54:22.702: INFO: Logging pods the kubelet thinks is on node i-0e1f2cd7cf96d3020 Mar 27 23:54:22.887: INFO: csi-mockplugin-attacher-0 started at 2023-03-27 23:52:09 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:22.887: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:54:22.887: INFO: csi-mockplugin-0 started at 2023-03-27 23:52:09 +0000 UTC (0+3 container statuses recorded) Mar 27 23:54:22.887: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:54:22.887: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:54:22.887: INFO: Container mock ready: true, restart count 0 Mar 27 23:54:22.887: INFO: netserver-3 started at 2023-03-27 23:54:08 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:22.887: INFO: Container webserver ready: true, restart count 0 Mar 27 23:54:22.887: INFO: up-down-1-hhmcz started at 2023-03-27 23:54:15 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:22.887: INFO: Container up-down-1 ready: true, restart count 0 Mar 27 23:54:22.887: INFO: coredns-559769c974-scn7m started at 2023-03-27 23:30:58 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:22.887: INFO: Container coredns ready: true, restart count 0 Mar 27 23:54:22.887: INFO: hostexec-i-0e1f2cd7cf96d3020-p7tdv started at 2023-03-27 23:54:03 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:22.887: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:54:22.887: INFO: csi-hostpathplugin-0 started at 2023-03-27 23:54:16 +0000 UTC (0+7 container statuses recorded) Mar 27 23:54:22.887: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:54:22.887: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:54:22.887: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:54:22.887: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 27 23:54:22.887: INFO: Container hostpath ready: true, restart count 0 Mar 27 23:54:22.887: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:54:22.887: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:54:22.887: INFO: pause-pod-1 started at 2023-03-27 23:53:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:22.887: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:54:22.887: INFO: ebs-csi-node-vcn4h started at 2023-03-27 23:22:28 +0000 UTC (0+3 container statuses recorded) Mar 27 23:54:22.887: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:54:22.887: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:54:22.887: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:54:22.887: INFO: pod-subpath-test-dynamicpv-kqsd started at 2023-03-27 23:54:20 +0000 UTC (2+1 container statuses recorded) Mar 27 23:54:22.887: INFO: Init container init-volume-dynamicpv-kqsd ready: false, restart count 0 Mar 27 23:54:22.887: INFO: Init container test-init-volume-dynamicpv-kqsd ready: false, restart count 0 Mar 27 23:54:22.887: INFO: Container test-container-subpath-dynamicpv-kqsd ready: false, restart count 0 Mar 27 23:54:22.887: INFO: kube-proxy-i-0e1f2cd7cf96d3020 started at 2023-03-27 23:22:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:22.887: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:54:22.887: INFO: calico-node-9fzr7 started at 2023-03-27 23:22:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:54:22.887: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:54:22.887: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:54:22.887: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:54:22.887: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:54:22.887: INFO: coredns-autoscaler-7cb5c5b969-2k8xf started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:22.887: INFO: Container autoscaler ready: true, restart count 0 Mar 27 23:54:23.563: INFO: Latency metrics for node i-0e1f2cd7cf96d3020 END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:54:23.563 (6.602s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - dump namespaces | framework.go:209 @ 03/27/23 23:54:23.563 (6.602s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/27/23 23:54:23.563 STEP: Destroying namespace "provisioning-2337" for this suite. - test/e2e/framework/framework.go:351 @ 03/27/23 23:54:23.563 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] subPath - tear down framework | framework.go:206 @ 03/27/23 23:54:23.741 (179ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:54:23.741 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:54:23.742 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(default\sfs\)\]\svolumes\sshould\sallow\sexec\sof\sfiles\son\sthe\svolume$'
[FAILED] failed to get logs from exec-volume-test-dynamicpv-f85l for exec-container-dynamicpv-f85l: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-9843/pods/exec-volume-test-dynamicpv-f85l/log?container=exec-container-dynamicpv-f85l&previous=false": stream error: stream ID 4435; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:56:59.022from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:56:38.707 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:56:38.707 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] volumes - set up framework | framework.go:191 @ 03/27/23 23:56:38.707 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/27/23 23:56:38.707 Mar 27 23:56:38.707: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename volume - test/e2e/framework/framework.go:250 @ 03/27/23 23:56:38.708 E0327 23:56:38.851570 7310 retrywatcher.go:130] "Watch failed" err="context canceled" STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/27/23 23:56:39.24 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/27/23 23:56:39.592 E0327 23:56:39.852414 7310 retrywatcher.go:130] "Watch failed" err="context canceled" < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] volumes - set up framework | framework.go:191 @ 03/27/23 23:56:39.944 (1.236s) > Enter [BeforeEach] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:56:39.944 < Exit [BeforeEach] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:56:39.944 (0s) > Enter [It] should allow exec of files on the volume - test/e2e/storage/testsuites/volumes.go:194 @ 03/27/23 23:56:39.944 Mar 27 23:56:39.944: INFO: Creating resource for dynamic PV Mar 27 23:56:39.944: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass volume-9843-e2e-scvcbtl - test/e2e/storage/framework/volume_resource.go:102 @ 03/27/23 23:56:39.944 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/27/23 23:56:40.132 Mar 27 23:56:40.132: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod exec-volume-test-dynamicpv-f85l - test/e2e/storage/testsuites/volumes.go:253 @ 03/27/23 23:56:40.486 STEP: Creating a pod to test exec-volume-test - test/e2e/framework/pod/output/output.go:243 @ 03/27/23 23:56:40.486 E0327 23:56:40.852870 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:41.852881 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:42.853384 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:43.854260 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:44.855230 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:45.855964 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:46.856019 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:47.857082 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:48.858095 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:49.858219 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:50.858765 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:51.859255 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:52.859692 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:53.860779 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:54.861565 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:55.862524 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:56.862526 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:56:57.863370 7310 retrywatcher.go:130] "Watch failed" err="context canceled" STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/27/23 23:56:58.287 Mar 27 23:56:58.464: INFO: Trying to get logs from node i-022003b3a45876d5d pod exec-volume-test-dynamicpv-f85l container exec-container-dynamicpv-f85l: <nil> Mar 27 23:56:58.640: INFO: Failed to get logs from node "i-022003b3a45876d5d" pod "exec-volume-test-dynamicpv-f85l" container "exec-container-dynamicpv-f85l". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-9843/pods/exec-volume-test-dynamicpv-f85l/log?container=exec-container-dynamicpv-f85l&previous=false": stream error: stream ID 4435; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/27/23 23:56:58.64 E0327 23:56:58.864334 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:56:59.021: INFO: Unexpected error: <*fmt.wrapError | 0xc0046252a0>: failed to get logs from exec-volume-test-dynamicpv-f85l for exec-container-dynamicpv-f85l: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-9843/pods/exec-volume-test-dynamicpv-f85l/log?container=exec-container-dynamicpv-f85l&previous=false": stream error: stream ID 4435; INTERNAL_ERROR; received from peer { msg: "failed to get logs from exec-volume-test-dynamicpv-f85l for exec-container-dynamicpv-f85l: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-9843/pods/exec-volume-test-dynamicpv-f85l/log?container=exec-container-dynamicpv-f85l&previous=false\": stream error: stream ID 4435; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc0014cecf0>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-9843/pods/exec-volume-test-dynamicpv-f85l/log?container=exec-container-dynamicpv-f85l&previous=false", Err: <http2.StreamError>{ StreamID: 4435, Code: 2, Cause: <*errors.errorString | 0xc0001e96a0>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from exec-volume-test-dynamicpv-f85l for exec-container-dynamicpv-f85l: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-9843/pods/exec-volume-test-dynamicpv-f85l/log?container=exec-container-dynamicpv-f85l&previous=false": stream error: stream ID 4435; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:56:59.022 < Exit [It] should allow exec of files on the volume - test/e2e/storage/testsuites/volumes.go:194 @ 03/27/23 23:56:59.022 (19.078s) > Enter [AfterEach] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:56:59.022 Mar 27 23:56:59.022: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:56:59.201 (180ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/storage/testsuites/volumes.go:197 @ 03/27/23 23:56:59.201 STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/27/23 23:56:59.201 Mar 27 23:56:59.555: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.com8nfxb" Mar 27 23:56:59.733: INFO: Waiting up to 5m0s for PersistentVolume pvc-99bb4103-fb34-4c51-a096-f615aab40855 to get deleted E0327 23:56:59.864865 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:56:59.910: INFO: PersistentVolume pvc-99bb4103-fb34-4c51-a096-f615aab40855 found and phase=Released (176.340462ms) E0327 23:57:00.865016 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:57:01.865918 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:57:02.866493 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:57:03.867045 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:57:04.867421 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:57:05.087: INFO: PersistentVolume pvc-99bb4103-fb34-4c51-a096-f615aab40855 found and phase=Released (5.353243633s) E0327 23:57:05.868505 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:57:06.868904 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:57:07.868959 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:57:08.870038 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:57:09.871027 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:57:10.264: INFO: PersistentVolume pvc-99bb4103-fb34-4c51-a096-f615aab40855 found and phase=Released (10.530129567s) E0327 23:57:10.871156 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:57:11.871543 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:57:12.872042 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:57:13.872885 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:57:14.873038 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:57:15.441: INFO: PersistentVolume pvc-99bb4103-fb34-4c51-a096-f615aab40855 was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/27/23 23:57:15.441 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/storage/testsuites/volumes.go:197 @ 03/27/23 23:57:15.622 (16.421s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:57:15.622 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:57:15.622 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - dump namespaces | framework.go:209 @ 03/27/23 23:57:15.622 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:57:15.623 STEP: Collecting events from namespace "volume-9843". - test/e2e/framework/debug/dump.go:42 @ 03/27/23 23:57:15.623 STEP: Found 9 events. - test/e2e/framework/debug/dump.go:46 @ 03/27/23 23:57:15.802 Mar 27 23:57:15.802: INFO: At 2023-03-27 23:56:40 +0000 UTC - event for ebs.csi.aws.com8nfxb: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 27 23:57:15.802: INFO: At 2023-03-27 23:56:40 +0000 UTC - event for ebs.csi.aws.com8nfxb: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 27 23:57:15.802: INFO: At 2023-03-27 23:56:40 +0000 UTC - event for ebs.csi.aws.com8nfxb: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } Provisioning: External provisioner is provisioning volume for claim "volume-9843/ebs.csi.aws.com8nfxb" Mar 27 23:57:15.802: INFO: At 2023-03-27 23:56:43 +0000 UTC - event for ebs.csi.aws.com8nfxb: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } ProvisioningSucceeded: Successfully provisioned volume pvc-99bb4103-fb34-4c51-a096-f615aab40855 Mar 27 23:57:15.802: INFO: At 2023-03-27 23:56:44 +0000 UTC - event for exec-volume-test-dynamicpv-f85l: {default-scheduler } Scheduled: Successfully assigned volume-9843/exec-volume-test-dynamicpv-f85l to i-022003b3a45876d5d Mar 27 23:57:15.802: INFO: At 2023-03-27 23:56:48 +0000 UTC - event for exec-volume-test-dynamicpv-f85l: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-99bb4103-fb34-4c51-a096-f615aab40855" Mar 27 23:57:15.802: INFO: At 2023-03-27 23:56:55 +0000 UTC - event for exec-volume-test-dynamicpv-f85l: {kubelet i-022003b3a45876d5d} Pulled: Container image "registry.k8s.io/e2e-test-images/nginx:1.14-4" already present on machine Mar 27 23:57:15.802: INFO: At 2023-03-27 23:56:55 +0000 UTC - event for exec-volume-test-dynamicpv-f85l: {kubelet i-022003b3a45876d5d} Created: Created container exec-container-dynamicpv-f85l Mar 27 23:57:15.802: INFO: At 2023-03-27 23:56:55 +0000 UTC - event for exec-volume-test-dynamicpv-f85l: {kubelet i-022003b3a45876d5d} Started: Started container exec-container-dynamicpv-f85l E0327 23:57:15.873834 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:57:15.985: INFO: POD NODE PHASE GRACE CONDITIONS Mar 27 23:57:15.985: INFO: Mar 27 23:57:16.162: INFO: Logging node info for node i-022003b3a45876d5d Mar 27 23:57:16.339: INFO: Node Info: &Node{ObjectMeta:{i-022003b3a45876d5d 20348b58-b610-41a0-accb-ca195a540336 63703 0 2023-03-27 23:33:26 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-022003b3a45876d5d kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-022003b3a45876d5d topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-capacity-9703":"i-022003b3a45876d5d","ebs.csi.aws.com":"i-022003b3a45876d5d"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.49.118/19 projectcalico.org/IPv4IPIPTunnelAddr:100.115.192.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:33:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:33:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:57:04 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:57:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-022003b3a45876d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:33:51 +0000 UTC,LastTransitionTime:2023-03-27 23:33:51 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:57:04 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:57:04 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:57:04 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:57:04 +0000 UTC,LastTransitionTime:2023-03-27 23:33:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.118,},NodeAddress{Type:ExternalIP,Address:54.180.103.89,},NodeAddress{Type:InternalDNS,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-103-89.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccb51411bbbb92946da93b9a452fe,SystemUUID:ec2ccb51-411b-bbb9-2946-da93b9a452fe,BootID:841d91a9-bfa5-4e0f-9279-7c83efeef8ca,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-0068da3fb4ca41012 kubernetes.io/csi/ebs.csi.aws.com^vol-027c7267daa9fbf92],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-027c7267daa9fbf92,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0068da3fb4ca41012,DevicePath:,},},Config:nil,},} Mar 27 23:57:16.339: INFO: Logging kubelet events for node i-022003b3a45876d5d Mar 27 23:57:16.522: INFO: Logging pods the kubelet thinks is on node i-022003b3a45876d5d Mar 27 23:57:16.705: INFO: kube-proxy-i-022003b3a45876d5d started at 2023-03-27 23:33:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:16.705: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:57:16.705: INFO: ebs-csi-node-2knd4 started at 2023-03-27 23:33:27 +0000 UTC (0+3 container statuses recorded) Mar 27 23:57:16.705: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:57:16.705: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:57:16.705: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:57:16.705: INFO: pod-e8ce79b8-5adf-437d-8106-7a7e161a7d45 started at 2023-03-27 23:57:11 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:16.705: INFO: Container test-container ready: false, restart count 0 Mar 27 23:57:16.705: INFO: dns-test-18d2793c-ef0d-4be1-b416-20c2d9b9d6d6 started at 2023-03-27 23:56:47 +0000 UTC (0+3 container statuses recorded) Mar 27 23:57:16.705: INFO: Container jessie-querier ready: true, restart count 0 Mar 27 23:57:16.705: INFO: Container querier ready: true, restart count 0 Mar 27 23:57:16.705: INFO: Container webserver ready: true, restart count 0 Mar 27 23:57:16.705: INFO: csi-mockplugin-0 started at 2023-03-27 23:57:09 +0000 UTC (0+3 container statuses recorded) Mar 27 23:57:16.705: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:57:16.705: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:57:16.705: INFO: Container mock ready: true, restart count 0 Mar 27 23:57:16.705: INFO: ss-0 started at 2023-03-27 23:54:46 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:16.705: INFO: Container webserver ready: true, restart count 0 Mar 27 23:57:16.705: INFO: netserver-0 started at 2023-03-27 23:56:45 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:16.705: INFO: Container webserver ready: true, restart count 0 Mar 27 23:57:16.705: INFO: pvc-volume-tester-dmzfq started at 2023-03-27 23:57:13 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:16.705: INFO: Container volume-tester ready: false, restart count 0 Mar 27 23:57:16.705: INFO: test-webserver-89bbdce1-151b-4393-83ca-9b6660221717 started at 2023-03-27 23:57:07 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:16.705: INFO: Container test-webserver ready: false, restart count 0 Mar 27 23:57:16.705: INFO: calico-node-2wjdb started at 2023-03-27 23:33:27 +0000 UTC (3+1 container statuses recorded) Mar 27 23:57:16.705: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:57:16.705: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:57:16.705: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:57:16.705: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:57:16.705: INFO: webserver-pod started at 2023-03-27 23:52:55 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:16.705: INFO: Container agnhost ready: false, restart count 0 Mar 27 23:57:16.705: INFO: netserver-0 started at 2023-03-27 23:56:49 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:16.705: INFO: Container webserver ready: true, restart count 0 Mar 27 23:57:16.705: INFO: ss-0 started at 2023-03-27 23:55:55 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:16.705: INFO: Container webserver ready: true, restart count 0 Mar 27 23:57:16.705: INFO: test-container-pod started at 2023-03-27 23:57:11 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:16.705: INFO: Container webserver ready: true, restart count 0 E0327 23:57:16.874063 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:57:17.322: INFO: Latency metrics for node i-022003b3a45876d5d Mar 27 23:57:17.322: INFO: Logging node info for node i-05ec0d9f3272223ec Mar 27 23:57:17.499: INFO: Node Info: &Node{ObjectMeta:{i-05ec0d9f3272223ec 365deda4-148e-4593-abdf-3103630fef28 63601 0 2023-03-27 23:25:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-05ec0d9f3272223ec kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-05ec0d9f3272223ec topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-05ec0d9f3272223ec"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.52.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.113.211.128 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:25:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:25:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:57:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-27 23:57:08 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-05ec0d9f3272223ec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:25:42 +0000 UTC,LastTransitionTime:2023-03-27 23:25:42 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:57:07 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:57:07 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:57:07 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:57:07 +0000 UTC,LastTransitionTime:2023-03-27 23:25:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.36,},NodeAddress{Type:ExternalIP,Address:3.34.125.198,},NodeAddress{Type:InternalDNS,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-34-125-198.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dea8f8b5e44e8fd6ef34a7ee9507f,SystemUUID:ec2dea8f-8b5e-44e8-fd6e-f34a7ee9507f,BootID:d41641dd-09ee-4029-b1ae-fb0f0a4ef62f,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d kubernetes.io/csi/ebs.csi.aws.com^vol-05176a95497f5f56e kubernetes.io/csi/ebs.csi.aws.com^vol-054fcfb0606bc608a kubernetes.io/csi/ebs.csi.aws.com^vol-07ca1cb1c91fd8c50 kubernetes.io/csi/ebs.csi.aws.com^vol-0b2e20d8d826aa327],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-054fcfb0606bc608a,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-05176a95497f5f56e,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-07ca1cb1c91fd8c50,DevicePath:,},},Config:nil,},} Mar 27 23:57:17.499: INFO: Logging kubelet events for node i-05ec0d9f3272223ec Mar 27 23:57:17.683: INFO: Logging pods the kubelet thinks is on node i-05ec0d9f3272223ec E0327 23:57:17.874111 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:57:17.880: INFO: host-test-container-pod started at 2023-03-27 23:57:09 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:17.880: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:57:17.880: INFO: host-test-container-pod started at 2023-03-27 23:57:11 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:17.880: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:57:17.880: INFO: kube-proxy-i-05ec0d9f3272223ec started at 2023-03-27 23:25:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:17.880: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:57:17.880: INFO: ss-2 started at 2023-03-27 23:55:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:17.880: INFO: Container webserver ready: false, restart count 0 Mar 27 23:57:17.880: INFO: termination-message-containerc0e1fb41-524c-4eb1-9ba3-ee802af54285 started at 2023-03-27 23:57:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:17.880: INFO: Container termination-message-container ready: false, restart count 0 Mar 27 23:57:17.880: INFO: pod-projected-configmaps-43df35b1-ca59-4b46-8b19-2eaf76096541 started at 2023-03-27 23:55:50 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:17.880: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:57:17.880: INFO: inline-volume-tester-kwwd2 started at 2023-03-27 23:43:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:17.880: INFO: Container csi-volume-tester ready: false, restart count 0 Mar 27 23:57:17.880: INFO: ss-2 started at 2023-03-27 23:56:30 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:17.880: INFO: Container webserver ready: true, restart count 0 Mar 27 23:57:17.880: INFO: test-ss-0 started at 2023-03-27 23:56:52 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:17.880: INFO: Container webserver ready: true, restart count 0 Mar 27 23:57:17.880: INFO: test-container-pod started at 2023-03-27 23:57:09 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:17.880: INFO: Container webserver ready: true, restart count 0 Mar 27 23:57:17.880: INFO: ebs-csi-node-gjw52 started at 2023-03-27 23:25:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:57:17.880: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:57:17.880: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:57:17.880: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:57:17.880: INFO: coredns-559769c974-m6hh6 started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:17.880: INFO: Container coredns ready: true, restart count 0 Mar 27 23:57:17.880: INFO: calico-node-5lq5m started at 2023-03-27 23:25:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:57:17.880: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:57:17.880: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:57:17.880: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:57:17.880: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:57:17.880: INFO: netserver-1 started at 2023-03-27 23:56:45 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:17.880: INFO: Container webserver ready: true, restart count 0 Mar 27 23:57:17.880: INFO: netserver-1 started at 2023-03-27 23:56:50 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:17.880: INFO: Container webserver ready: true, restart count 0 Mar 27 23:57:18.518: INFO: Latency metrics for node i-05ec0d9f3272223ec Mar 27 23:57:18.518: INFO: Logging node info for node i-0b266cca2f40af1c1 Mar 27 23:57:18.695: INFO: Node Info: &Node{ObjectMeta:{i-0b266cca2f40af1c1 5f957448-9afc-4966-b9d6-82332b7a159d 63726 0 2023-03-27 23:29:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0b266cca2f40af1c1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0b266cca2f40af1c1 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-capacity-7422":"i-0b266cca2f40af1c1","csi-mock-csi-mock-volumes-expansion-7488":"csi-mock-csi-mock-volumes-expansion-7488","ebs.csi.aws.com":"i-0b266cca2f40af1c1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.116.128.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:30:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:30:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:56:20 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:57:11 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0b266cca2f40af1c1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:30:13 +0000 UTC,LastTransitionTime:2023-03-27 23:30:13 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:56:20 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:56:20 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:56:20 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:56:20 +0000 UTC,LastTransitionTime:2023-03-27 23:30:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.36,},NodeAddress{Type:ExternalIP,Address:13.209.22.95,},NodeAddress{Type:InternalDNS,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-209-22-95.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec247601b7b315369bfb310559a7772d,SystemUUID:ec247601-b7b3-1536-9bfb-310559a7772d,BootID:487b59c3-9394-4a90-a6ec-b20c4819fe9e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-7488^4 kubernetes.io/csi/ebs.csi.aws.com^vol-045230ea060c16d82],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-7488^4,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-045230ea060c16d82,DevicePath:,},},Config:nil,},} Mar 27 23:57:18.695: INFO: Logging kubelet events for node i-0b266cca2f40af1c1 E0327 23:57:18.874505 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:57:18.882: INFO: Logging pods the kubelet thinks is on node i-0b266cca2f40af1c1 Mar 27 23:57:19.069: INFO: pod-subpath-test-configmap-2sr8 started at 2023-03-27 23:56:33 +0000 UTC (1+2 container statuses recorded) Mar 27 23:57:19.069: INFO: Init container init-volume-configmap-2sr8 ready: true, restart count 0 Mar 27 23:57:19.069: INFO: Container test-container-subpath-configmap-2sr8 ready: true, restart count 3 Mar 27 23:57:19.069: INFO: Container test-container-volume-configmap-2sr8 ready: true, restart count 0 Mar 27 23:57:19.069: INFO: pod-ce82626b-b16a-482b-ab9b-950eef713c48 started at 2023-03-27 23:57:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container write-pod ready: false, restart count 0 Mar 27 23:57:19.069: INFO: csi-mockplugin-0 started at 2023-03-27 23:57:08 +0000 UTC (0+3 container statuses recorded) Mar 27 23:57:19.069: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:57:19.069: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:57:19.069: INFO: Container mock ready: true, restart count 0 Mar 27 23:57:19.069: INFO: pod-disruption-failure-ignore-0-g5fss started at 2023-03-27 23:56:49 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container c ready: true, restart count 0 Mar 27 23:57:19.069: INFO: csi-mockplugin-0 started at 2023-03-27 23:55:58 +0000 UTC (0+4 container statuses recorded) Mar 27 23:57:19.069: INFO: Container busybox ready: true, restart count 0 Mar 27 23:57:19.069: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:57:19.069: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:57:19.069: INFO: Container mock ready: true, restart count 0 Mar 27 23:57:19.069: INFO: ss-1 started at 2023-03-27 23:56:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container webserver ready: true, restart count 0 Mar 27 23:57:19.069: INFO: netserver-2 started at 2023-03-27 23:56:45 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container webserver ready: true, restart count 0 Mar 27 23:57:19.069: INFO: pod-disruption-failure-ignore-1-f68kw started at 2023-03-27 23:56:49 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container c ready: false, restart count 0 Mar 27 23:57:19.069: INFO: csi-mockplugin-attacher-0 started at 2023-03-27 23:57:09 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:57:19.069: INFO: hostexec-i-0b266cca2f40af1c1-j2xz5 started at 2023-03-27 23:57:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:57:19.069: INFO: kube-proxy-i-0b266cca2f40af1c1 started at 2023-03-27 23:29:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:57:19.069: INFO: calico-node-nmfrn started at 2023-03-27 23:29:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:57:19.069: INFO: Init container install-cni ready: true, restart count 1 Mar 27 23:57:19.069: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:57:19.069: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:57:19.069: INFO: pvc-volume-tester-pnxvr started at 2023-03-27 23:57:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container volume-tester ready: false, restart count 0 Mar 27 23:57:19.069: INFO: pvc-volume-tester-jrpzt started at 2023-03-27 23:56:12 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container volume-tester ready: true, restart count 0 Mar 27 23:57:19.069: INFO: csi-mockplugin-attacher-0 started at 2023-03-27 23:55:58 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:57:19.069: INFO: pod-1afcb14d-5380-47f1-8478-ae9b54880675 started at 2023-03-27 23:57:13 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container write-pod ready: true, restart count 0 Mar 27 23:57:19.069: INFO: pod-disruption-failure-ignore-0-c26nd started at 2023-03-27 23:57:03 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container c ready: false, restart count 0 Mar 27 23:57:19.069: INFO: pod-disruption-failure-ignore-3-826zs started at 2023-03-27 23:56:56 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container c ready: false, restart count 0 Mar 27 23:57:19.069: INFO: hostexec-i-0b266cca2f40af1c1-kl7lg started at 2023-03-27 23:57:07 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:57:19.069: INFO: ebs-csi-node-ztjm9 started at 2023-03-27 23:29:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:57:19.069: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:57:19.069: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:57:19.069: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:57:19.069: INFO: netserver-2 started at 2023-03-27 23:56:50 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container webserver ready: true, restart count 0 Mar 27 23:57:19.069: INFO: pod-disruption-failure-ignore-2-k4brl started at 2023-03-27 23:56:53 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:19.069: INFO: Container c ready: false, restart count 0 Mar 27 23:57:19.766: INFO: Latency metrics for node i-0b266cca2f40af1c1 Mar 27 23:57:19.766: INFO: Logging node info for node i-0d2b1184f62de758a E0327 23:57:19.875178 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:57:19.943: INFO: Node Info: &Node{ObjectMeta:{i-0d2b1184f62de758a 2e42ccb1-9580-4126-b5d6-05a5f98403c5 60152 0 2023-03-27 23:17:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d2b1184f62de758a kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d2b1184f62de758a"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.6/19 projectcalico.org/IPv4IPIPTunnelAddr:100.120.150.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-27 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-27 23:17:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:18:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:55:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0d2b1184f62de758a,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:18:49 +0000 UTC,LastTransitionTime:2023-03-27 23:18:49 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:55:58 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:55:58 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:55:58 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:55:58 +0000 UTC,LastTransitionTime:2023-03-27 23:18:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.6,},NodeAddress{Type:ExternalIP,Address:3.35.11.126,},NodeAddress{Type:InternalDNS,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-35-11-126.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2781baf4898d15db3c853bf5b6d100,SystemUUID:ec2781ba-f489-8d15-db3c-853bf5b6d100,BootID:5e905df0-494e-4050-b6d7-400763c3dd66,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45208814,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43629012,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074727,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:57:19.944: INFO: Logging kubelet events for node i-0d2b1184f62de758a Mar 27 23:57:20.134: INFO: Logging pods the kubelet thinks is on node i-0d2b1184f62de758a Mar 27 23:57:20.317: INFO: etcd-manager-main-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:57:20.317: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:57:20.317: INFO: kube-apiserver-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+2 container statuses recorded) Mar 27 23:57:20.317: INFO: Container healthcheck ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Container kube-apiserver ready: true, restart count 3 Mar 27 23:57:20.317: INFO: kube-controller-manager-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:20.317: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 27 23:57:20.317: INFO: kube-proxy-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:20.317: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:57:20.317: INFO: aws-cloud-controller-manager-776ld started at 2023-03-27 23:18:28 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:20.317: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 27 23:57:20.317: INFO: calico-kube-controllers-55f488656-vj6gh started at 2023-03-27 23:18:57 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:20.317: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 27 23:57:20.317: INFO: ebs-csi-controller-79f984545f-42vd6 started at 2023-03-27 23:19:48 +0000 UTC (0+5 container statuses recorded) Mar 27 23:57:20.317: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:57:20.317: INFO: etcd-manager-events-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:57:20.317: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:57:20.317: INFO: dns-controller-69987775c6-rzzjj started at 2023-03-27 23:18:25 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:20.317: INFO: Container dns-controller ready: true, restart count 0 Mar 27 23:57:20.317: INFO: calico-node-scbv2 started at 2023-03-27 23:18:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:57:20.317: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:57:20.317: INFO: ebs-csi-node-nvqcx started at 2023-03-27 23:18:29 +0000 UTC (0+3 container statuses recorded) Mar 27 23:57:20.317: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:57:20.317: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:57:20.317: INFO: kops-controller-w4b7t started at 2023-03-27 23:18:29 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:20.317: INFO: Container kops-controller ready: true, restart count 0 Mar 27 23:57:20.317: INFO: kube-scheduler-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:20.317: INFO: Container kube-scheduler ready: true, restart count 0 E0327 23:57:20.876009 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:57:20.972: INFO: Latency metrics for node i-0d2b1184f62de758a Mar 27 23:57:20.972: INFO: Logging node info for node i-0e1f2cd7cf96d3020 Mar 27 23:57:21.148: INFO: Node Info: &Node{ObjectMeta:{i-0e1f2cd7cf96d3020 efcf1e66-db0e-4256-a849-b9c377eef04e 63970 0 2023-03-27 23:22:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e1f2cd7cf96d3020 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0e1f2cd7cf96d3020 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-558":"i-0e1f2cd7cf96d3020","ebs.csi.aws.com":"i-0e1f2cd7cf96d3020"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.120/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.86.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:22:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:22:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:57:19 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0e1f2cd7cf96d3020,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:22:52 +0000 UTC,LastTransitionTime:2023-03-27 23:22:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:57:19 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:57:19 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:57:19 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:57:19 +0000 UTC,LastTransitionTime:2023-03-27 23:22:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.120,},NodeAddress{Type:ExternalIP,Address:15.164.224.157,},NodeAddress{Type:InternalDNS,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-164-224-157.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29484c87c97fb3d8a52788d5971660,SystemUUID:ec29484c-87c9-7fb3-d8a5-2788d5971660,BootID:b73c540f-bd49-470a-a7b3-08c31cc8c2fd,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonroot@sha256:ee9f50b3c64b174d296d91ca9f69a914ac30e59095dfb462b2b518ad28a63655 registry.k8s.io/e2e-test-images/nonroot:1.4],SizeBytes:17747885,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:57:21.148: INFO: Logging kubelet events for node i-0e1f2cd7cf96d3020 Mar 27 23:57:21.361: INFO: Logging pods the kubelet thinks is on node i-0e1f2cd7cf96d3020 Mar 27 23:57:21.565: INFO: ebs-csi-node-vcn4h started at 2023-03-27 23:22:28 +0000 UTC (0+3 container statuses recorded) Mar 27 23:57:21.565: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:57:21.565: INFO: csi-mockplugin-0 started at 2023-03-27 23:55:53 +0000 UTC (0+4 container statuses recorded) Mar 27 23:57:21.565: INFO: Container busybox ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Container mock ready: true, restart count 0 Mar 27 23:57:21.565: INFO: kube-proxy-i-0e1f2cd7cf96d3020 started at 2023-03-27 23:22:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:21.565: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:57:21.565: INFO: calico-node-9fzr7 started at 2023-03-27 23:22:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:57:21.565: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:57:21.565: INFO: coredns-autoscaler-7cb5c5b969-2k8xf started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:21.565: INFO: Container autoscaler ready: true, restart count 0 Mar 27 23:57:21.565: INFO: netserver-3 started at 2023-03-27 23:56:45 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:21.565: INFO: Container webserver ready: true, restart count 0 Mar 27 23:57:21.565: INFO: csi-mockplugin-0 started at 2023-03-27 23:55:53 +0000 UTC (0+4 container statuses recorded) Mar 27 23:57:21.565: INFO: Container busybox ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Container mock ready: true, restart count 0 Mar 27 23:57:21.565: INFO: csi-hostpathplugin-0 started at 2023-03-27 23:56:45 +0000 UTC (0+7 container statuses recorded) Mar 27 23:57:21.565: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Container hostpath ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:57:21.565: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:57:21.565: INFO: coredns-559769c974-scn7m started at 2023-03-27 23:30:58 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:21.565: INFO: Container coredns ready: true, restart count 0 Mar 27 23:57:21.565: INFO: netserver-3 started at 2023-03-27 23:56:50 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:21.565: INFO: Container webserver ready: false, restart count 0 Mar 27 23:57:21.565: INFO: inline-volume-tester-lw775 started at 2023-03-27 23:56:46 +0000 UTC (0+1 container statuses recorded) Mar 27 23:57:21.565: INFO: Container csi-volume-tester ready: true, restart count 0 E0327 23:57:21.876612 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:57:22.222: INFO: Latency metrics for node i-0e1f2cd7cf96d3020 END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:57:22.222 (6.599s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - dump namespaces | framework.go:209 @ 03/27/23 23:57:22.222 (6.599s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - tear down framework | framework.go:206 @ 03/27/23 23:57:22.222 STEP: Destroying namespace "volume-9843" for this suite. - test/e2e/framework/framework.go:351 @ 03/27/23 23:57:22.222 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (default fs)] volumes - tear down framework | framework.go:206 @ 03/27/23 23:57:22.4 (179ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:57:22.4 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:57:22.4 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sDynamic\sPV\s\(ext4\)\]\svolumes\sshould\sallow\sexec\sof\sfiles\son\sthe\svolume$'
[FAILED] failed to get logs from exec-volume-test-dynamicpv-4cmd for exec-container-dynamicpv-4cmd: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-8869/pods/exec-volume-test-dynamicpv-4cmd/log?container=exec-container-dynamicpv-4cmd&previous=false": stream error: stream ID 3865; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:54:35.236from junit_01.xml
> Enter [BeforeEach] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:54:14.937 < Exit [BeforeEach] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:54:14.937 (0s) > Enter [BeforeEach] [Testpattern: Dynamic PV (ext4)] volumes - set up framework | framework.go:191 @ 03/27/23 23:54:14.937 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/27/23 23:54:14.937 Mar 27 23:54:14.937: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename volume - test/e2e/framework/framework.go:250 @ 03/27/23 23:54:14.938 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/27/23 23:54:15.469 E0327 23:54:15.772131 7310 retrywatcher.go:130] "Watch failed" err="context canceled" STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/27/23 23:54:15.823 < Exit [BeforeEach] [Testpattern: Dynamic PV (ext4)] volumes - set up framework | framework.go:191 @ 03/27/23 23:54:16.176 (1.239s) > Enter [BeforeEach] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:54:16.176 < Exit [BeforeEach] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:54:16.176 (0s) > Enter [It] should allow exec of files on the volume - test/e2e/storage/testsuites/volumes.go:194 @ 03/27/23 23:54:16.176 Mar 27 23:54:16.176: INFO: Creating resource for dynamic PV Mar 27 23:54:16.176: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} STEP: creating a StorageClass volume-8869-e2e-scg4wv8 - test/e2e/storage/framework/volume_resource.go:102 @ 03/27/23 23:54:16.177 STEP: creating a claim - test/e2e/storage/framework/volume_resource.go:294 @ 03/27/23 23:54:16.355 Mar 27 23:54:16.355: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil STEP: Creating pod exec-volume-test-dynamicpv-4cmd - test/e2e/storage/testsuites/volumes.go:253 @ 03/27/23 23:54:16.726 STEP: Creating a pod to test exec-volume-test - test/e2e/framework/pod/output/output.go:243 @ 03/27/23 23:54:16.726 E0327 23:54:16.773255 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:17.774258 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:18.775330 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:19.775481 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:20.776112 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:21.777137 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:22.777999 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:23.778551 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:24.779534 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:25.779680 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:26.780089 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:27.780394 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:28.781420 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:29.781594 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:30.781790 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:31.782404 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:32.783229 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:33.783421 7310 retrywatcher.go:130] "Watch failed" err="context canceled" STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/27/23 23:54:34.516 Mar 27 23:54:34.692: INFO: Trying to get logs from node i-022003b3a45876d5d pod exec-volume-test-dynamicpv-4cmd container exec-container-dynamicpv-4cmd: <nil> E0327 23:54:34.783777 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:54:34.868: INFO: Failed to get logs from node "i-022003b3a45876d5d" pod "exec-volume-test-dynamicpv-4cmd" container "exec-container-dynamicpv-4cmd". Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-8869/pods/exec-volume-test-dynamicpv-4cmd/log?container=exec-container-dynamicpv-4cmd&previous=false": stream error: stream ID 3865; INTERNAL_ERROR; received from peer STEP: delete the pod - test/e2e/framework/pod/output/output.go:168 @ 03/27/23 23:54:34.868 Mar 27 23:54:35.235: INFO: Unexpected error: <*fmt.wrapError | 0xc00532d240>: failed to get logs from exec-volume-test-dynamicpv-4cmd for exec-container-dynamicpv-4cmd: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-8869/pods/exec-volume-test-dynamicpv-4cmd/log?container=exec-container-dynamicpv-4cmd&previous=false": stream error: stream ID 3865; INTERNAL_ERROR; received from peer { msg: "failed to get logs from exec-volume-test-dynamicpv-4cmd for exec-container-dynamicpv-4cmd: Get \"https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-8869/pods/exec-volume-test-dynamicpv-4cmd/log?container=exec-container-dynamicpv-4cmd&previous=false\": stream error: stream ID 3865; INTERNAL_ERROR; received from peer", err: <*url.Error | 0xc0028432f0>{ Op: "Get", URL: "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-8869/pods/exec-volume-test-dynamicpv-4cmd/log?container=exec-container-dynamicpv-4cmd&previous=false", Err: <http2.StreamError>{ StreamID: 3865, Code: 2, Cause: <*errors.errorString | 0xc0001e96a0>{ s: "received from peer", }, }, }, } [FAILED] failed to get logs from exec-volume-test-dynamicpv-4cmd for exec-container-dynamicpv-4cmd: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/volume-8869/pods/exec-volume-test-dynamicpv-4cmd/log?container=exec-container-dynamicpv-4cmd&previous=false": stream error: stream ID 3865; INTERNAL_ERROR; received from peer In [It] at: test/e2e/framework/pod/output/output.go:247 @ 03/27/23 23:54:35.236 < Exit [It] should allow exec of files on the volume - test/e2e/storage/testsuites/volumes.go:194 @ 03/27/23 23:54:35.236 (19.059s) > Enter [AfterEach] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:54:35.236 Mar 27 23:54:35.236: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:54:35.413 (178ms) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/storage/testsuites/volumes.go:197 @ 03/27/23 23:54:35.413 STEP: Deleting pvc - test/e2e/storage/framework/volume_resource.go:181 @ 03/27/23 23:54:35.413 Mar 27 23:54:35.767: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.combxzmb" E0327 23:54:35.783860 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:54:35.947: INFO: Waiting up to 5m0s for PersistentVolume pvc-e7236b85-684b-408d-aa53-f721b803cf3e to get deleted Mar 27 23:54:36.123: INFO: PersistentVolume pvc-e7236b85-684b-408d-aa53-f721b803cf3e found and phase=Released (176.297925ms) E0327 23:54:36.784906 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:37.785017 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:38.785930 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:39.786064 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:40.786775 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:54:41.300: INFO: PersistentVolume pvc-e7236b85-684b-408d-aa53-f721b803cf3e found and phase=Released (5.35334152s) E0327 23:54:41.786902 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:42.787049 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:43.787219 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:44.787350 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:45.787479 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:54:46.478: INFO: PersistentVolume pvc-e7236b85-684b-408d-aa53-f721b803cf3e found and phase=Released (10.531093617s) E0327 23:54:46.787893 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:47.788374 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:48.789006 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:49.789187 7310 retrywatcher.go:130] "Watch failed" err="context canceled" E0327 23:54:50.789633 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:54:51.655: INFO: PersistentVolume pvc-e7236b85-684b-408d-aa53-f721b803cf3e was removed STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/27/23 23:54:51.655 E0327 23:54:51.790478 7310 retrywatcher.go:130] "Watch failed" err="context canceled" < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/storage/testsuites/volumes.go:197 @ 03/27/23 23:54:51.833 (16.419s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:54:51.833 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:54:51.833 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - dump namespaces | framework.go:209 @ 03/27/23 23:54:51.833 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:54:51.833 STEP: Collecting events from namespace "volume-8869". - test/e2e/framework/debug/dump.go:42 @ 03/27/23 23:54:51.833 STEP: Found 9 events. - test/e2e/framework/debug/dump.go:46 @ 03/27/23 23:54:52.01 Mar 27 23:54:52.010: INFO: At 2023-03-27 23:54:16 +0000 UTC - event for ebs.csi.aws.combxzmb: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding Mar 27 23:54:52.010: INFO: At 2023-03-27 23:54:16 +0000 UTC - event for ebs.csi.aws.combxzmb: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 27 23:54:52.010: INFO: At 2023-03-27 23:54:16 +0000 UTC - event for ebs.csi.aws.combxzmb: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } Provisioning: External provisioner is provisioning volume for claim "volume-8869/ebs.csi.aws.combxzmb" Mar 27 23:54:52.010: INFO: At 2023-03-27 23:54:20 +0000 UTC - event for ebs.csi.aws.combxzmb: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } ProvisioningSucceeded: Successfully provisioned volume pvc-e7236b85-684b-408d-aa53-f721b803cf3e Mar 27 23:54:52.010: INFO: At 2023-03-27 23:54:20 +0000 UTC - event for exec-volume-test-dynamicpv-4cmd: {default-scheduler } Scheduled: Successfully assigned volume-8869/exec-volume-test-dynamicpv-4cmd to i-022003b3a45876d5d Mar 27 23:54:52.010: INFO: At 2023-03-27 23:54:28 +0000 UTC - event for exec-volume-test-dynamicpv-4cmd: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-e7236b85-684b-408d-aa53-f721b803cf3e" Mar 27 23:54:52.010: INFO: At 2023-03-27 23:54:31 +0000 UTC - event for exec-volume-test-dynamicpv-4cmd: {kubelet i-022003b3a45876d5d} Pulled: Container image "registry.k8s.io/e2e-test-images/nginx:1.14-4" already present on machine Mar 27 23:54:52.010: INFO: At 2023-03-27 23:54:31 +0000 UTC - event for exec-volume-test-dynamicpv-4cmd: {kubelet i-022003b3a45876d5d} Created: Created container exec-container-dynamicpv-4cmd Mar 27 23:54:52.010: INFO: At 2023-03-27 23:54:31 +0000 UTC - event for exec-volume-test-dynamicpv-4cmd: {kubelet i-022003b3a45876d5d} Started: Started container exec-container-dynamicpv-4cmd Mar 27 23:54:52.186: INFO: POD NODE PHASE GRACE CONDITIONS Mar 27 23:54:52.186: INFO: Mar 27 23:54:52.363: INFO: Logging node info for node i-022003b3a45876d5d Mar 27 23:54:52.539: INFO: Node Info: &Node{ObjectMeta:{i-022003b3a45876d5d 20348b58-b610-41a0-accb-ca195a540336 56891 0 2023-03-27 23:33:26 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-022003b3a45876d5d kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-022003b3a45876d5d topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-022003b3a45876d5d"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.49.118/19 projectcalico.org/IPv4IPIPTunnelAddr:100.115.192.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:33:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:33:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:54:48 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:54:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-022003b3a45876d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:33:51 +0000 UTC,LastTransitionTime:2023-03-27 23:33:51 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:50 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:50 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:50 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:54:50 +0000 UTC,LastTransitionTime:2023-03-27 23:33:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.118,},NodeAddress{Type:ExternalIP,Address:54.180.103.89,},NodeAddress{Type:InternalDNS,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-103-89.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccb51411bbbb92946da93b9a452fe,SystemUUID:ec2ccb51-411b-bbb9-2946-da93b9a452fe,BootID:841d91a9-bfa5-4e0f-9279-7c83efeef8ca,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-027c7267daa9fbf92],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-027c7267daa9fbf92,DevicePath:,},},Config:nil,},} Mar 27 23:54:52.539: INFO: Logging kubelet events for node i-022003b3a45876d5d Mar 27 23:54:52.738: INFO: Logging pods the kubelet thinks is on node i-022003b3a45876d5d E0327 23:54:52.791306 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:54:52.919: INFO: pfpod started at 2023-03-27 23:53:13 +0000 UTC (0+2 container statuses recorded) Mar 27 23:54:52.919: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:54:52.919: INFO: Container readiness ready: false, restart count 0 Mar 27 23:54:52.919: INFO: ss-0 started at 2023-03-27 23:54:46 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:52.919: INFO: Container webserver ready: true, restart count 0 Mar 27 23:54:52.919: INFO: up-down-2-jwgr8 started at 2023-03-27 23:54:19 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:52.919: INFO: Container up-down-2 ready: true, restart count 0 Mar 27 23:54:52.919: INFO: calico-node-2wjdb started at 2023-03-27 23:33:27 +0000 UTC (3+1 container statuses recorded) Mar 27 23:54:52.919: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:54:52.919: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:54:52.919: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:54:52.919: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:54:52.919: INFO: webserver-pod started at 2023-03-27 23:52:55 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:52.919: INFO: Container agnhost ready: false, restart count 0 Mar 27 23:54:52.919: INFO: evicted-pod started at 2023-03-27 23:54:31 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:52.919: INFO: Container evicted-pod ready: true, restart count 0 Mar 27 23:54:52.919: INFO: netserver-0 started at 2023-03-27 23:54:44 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:52.919: INFO: Container webserver ready: false, restart count 0 Mar 27 23:54:52.919: INFO: pfpod started at 2023-03-27 23:52:53 +0000 UTC (0+2 container statuses recorded) Mar 27 23:54:52.919: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:54:52.919: INFO: Container readiness ready: false, restart count 0 Mar 27 23:54:52.919: INFO: kube-proxy-i-022003b3a45876d5d started at 2023-03-27 23:33:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:52.919: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:54:52.919: INFO: ebs-csi-node-2knd4 started at 2023-03-27 23:33:27 +0000 UTC (0+3 container statuses recorded) Mar 27 23:54:52.919: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:54:52.919: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:54:52.919: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:54:53.528: INFO: Latency metrics for node i-022003b3a45876d5d Mar 27 23:54:53.528: INFO: Logging node info for node i-05ec0d9f3272223ec Mar 27 23:54:53.709: INFO: Node Info: &Node{ObjectMeta:{i-05ec0d9f3272223ec 365deda4-148e-4593-abdf-3103630fef28 55986 0 2023-03-27 23:25:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-05ec0d9f3272223ec kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-05ec0d9f3272223ec topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-05ec0d9f3272223ec"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.52.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.113.211.128 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:25:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:25:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:54:18 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:54:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-05ec0d9f3272223ec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:25:42 +0000 UTC,LastTransitionTime:2023-03-27 23:25:42 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:24 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:24 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:24 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:54:24 +0000 UTC,LastTransitionTime:2023-03-27 23:25:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.36,},NodeAddress{Type:ExternalIP,Address:3.34.125.198,},NodeAddress{Type:InternalDNS,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-34-125-198.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dea8f8b5e44e8fd6ef34a7ee9507f,SystemUUID:ec2dea8f-8b5e-44e8-fd6e-f34a7ee9507f,BootID:d41641dd-09ee-4029-b1ae-fb0f0a4ef62f,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d kubernetes.io/csi/ebs.csi.aws.com^vol-0685d11994819f57d],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-0685d11994819f57d,DevicePath:,},},Config:nil,},} Mar 27 23:54:53.709: INFO: Logging kubelet events for node i-05ec0d9f3272223ec E0327 23:54:53.792232 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:54:53.907: INFO: Logging pods the kubelet thinks is on node i-05ec0d9f3272223ec Mar 27 23:54:54.090: INFO: calico-node-5lq5m started at 2023-03-27 23:25:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:54:54.090: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:54:54.090: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:54:54.090: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:54:54.090: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:54:54.090: INFO: execpodnrx8g started at 2023-03-27 23:54:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:54.090: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:54:54.090: INFO: verify-service-up-host-exec-pod started at 2023-03-27 23:54:52 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:54.090: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:54:54.090: INFO: hostpath-symlink-prep-provisioning-2089 started at 2023-03-27 23:54:53 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:54.090: INFO: Container init-volume-provisioning-2089 ready: false, restart count 0 Mar 27 23:54:54.090: INFO: httpd started at 2023-03-27 23:54:23 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:54.090: INFO: Container httpd ready: true, restart count 0 Mar 27 23:54:54.090: INFO: hostexec-i-05ec0d9f3272223ec-vvhbs started at 2023-03-27 23:54:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:54.090: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:54:54.090: INFO: kube-proxy-i-05ec0d9f3272223ec started at 2023-03-27 23:25:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:54.090: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:54:54.090: INFO: up-down-2-jsqpk started at 2023-03-27 23:54:19 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:54.090: INFO: Container up-down-2 ready: true, restart count 0 Mar 27 23:54:54.090: INFO: netserver-1 started at 2023-03-27 23:54:44 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:54.090: INFO: Container webserver ready: false, restart count 0 Mar 27 23:54:54.090: INFO: ebs-csi-node-gjw52 started at 2023-03-27 23:25:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:54:54.090: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:54:54.090: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:54:54.090: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:54:54.090: INFO: coredns-559769c974-m6hh6 started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:54.090: INFO: Container coredns ready: true, restart count 0 Mar 27 23:54:54.090: INFO: inline-volume-tester-kwwd2 started at 2023-03-27 23:43:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:54.090: INFO: Container csi-volume-tester ready: false, restart count 0 Mar 27 23:54:54.090: INFO: pod-ephm-test-secret-jtp6 started at 2023-03-27 23:53:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:54.090: INFO: Container test-container-subpath-secret-jtp6 ready: false, restart count 0 Mar 27 23:54:54.090: INFO: pod1 started at 2023-03-27 23:53:57 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:54.090: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:54:54.715: INFO: Latency metrics for node i-05ec0d9f3272223ec Mar 27 23:54:54.715: INFO: Logging node info for node i-0b266cca2f40af1c1 E0327 23:54:54.793034 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:54:54.892: INFO: Node Info: &Node{ObjectMeta:{i-0b266cca2f40af1c1 5f957448-9afc-4966-b9d6-82332b7a159d 57008 0 2023-03-27 23:29:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0b266cca2f40af1c1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0b266cca2f40af1c1 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7886":"i-0b266cca2f40af1c1","csi-mock-csi-mock-volumes-attach-8822":"i-0b266cca2f40af1c1","csi-mock-csi-mock-volumes-expansion-2991":"i-0b266cca2f40af1c1","ebs.csi.aws.com":"i-0b266cca2f40af1c1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.116.128.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:30:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:30:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:54:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2023-03-27 23:54:54 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0b266cca2f40af1c1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:30:13 +0000 UTC,LastTransitionTime:2023-03-27 23:30:13 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:48 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:48 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:48 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:54:48 +0000 UTC,LastTransitionTime:2023-03-27 23:30:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.36,},NodeAddress{Type:ExternalIP,Address:13.209.22.95,},NodeAddress{Type:InternalDNS,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-209-22-95.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec247601b7b315369bfb310559a7772d,SystemUUID:ec247601-b7b3-1536-9bfb-310559a7772d,BootID:487b59c3-9394-4a90-a6ec-b20c4819fe9e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7886^c04b2cb6-ccfa-11ed-a1b1-fa34c4e6c5a5 kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-2991^9ab3bf37-ccfa-11ed-a28d-ead8e583275f kubernetes.io/csi/ebs.csi.aws.com^vol-02340833e64aefd4f],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-2991^9ab3bf37-ccfa-11ed-a28d-ead8e583275f,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7886^c04b2cb6-ccfa-11ed-a1b1-fa34c4e6c5a5,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-02340833e64aefd4f,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7886^c4676d58-ccfa-11ed-a1b1-fa34c4e6c5a5,DevicePath:,},},Config:nil,},} Mar 27 23:54:54.892: INFO: Logging kubelet events for node i-0b266cca2f40af1c1 Mar 27 23:54:55.076: INFO: Logging pods the kubelet thinks is on node i-0b266cca2f40af1c1 Mar 27 23:54:55.265: INFO: csi-mockplugin-0 started at 2023-03-27 23:54:51 +0000 UTC (0+3 container statuses recorded) Mar 27 23:54:55.265: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:54:55.265: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:54:55.265: INFO: Container mock ready: true, restart count 0 Mar 27 23:54:55.265: INFO: csi-mockplugin-0 started at 2023-03-27 23:53:39 +0000 UTC (0+3 container statuses recorded) Mar 27 23:54:55.265: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:54:55.265: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:54:55.265: INFO: Container mock ready: true, restart count 0 Mar 27 23:54:55.265: INFO: csi-mockplugin-resizer-0 started at 2023-03-27 23:53:40 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:55.265: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:54:55.265: INFO: csi-mockplugin-attacher-0 started at 2023-03-27 23:53:39 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:55.265: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:54:55.265: INFO: pvc-volume-tester-kjswl started at 2023-03-27 23:53:43 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:55.265: INFO: Container volume-tester ready: true, restart count 0 Mar 27 23:54:55.265: INFO: csi-hostpathplugin-0 started at 2023-03-27 23:54:43 +0000 UTC (0+7 container statuses recorded) Mar 27 23:54:55.265: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:54:55.265: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:54:55.265: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:54:55.265: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 27 23:54:55.265: INFO: Container hostpath ready: true, restart count 0 Mar 27 23:54:55.265: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:54:55.265: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:54:55.265: INFO: hostexec-i-0b266cca2f40af1c1-xcn9p started at 2023-03-27 23:54:47 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:55.265: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:54:55.265: INFO: ebs-csi-node-ztjm9 started at 2023-03-27 23:29:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:54:55.265: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:54:55.265: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:54:55.265: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:54:55.265: INFO: inline-volume-tester2-4j4zs started at 2023-03-27 23:54:53 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:55.265: INFO: Container csi-volume-tester ready: false, restart count 0 Mar 27 23:54:55.265: INFO: netserver-2 started at 2023-03-27 23:54:44 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:55.265: INFO: Container webserver ready: false, restart count 0 Mar 27 23:54:55.265: INFO: pvc-volume-tester-vmrlf started at 2023-03-27 23:54:55 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:55.265: INFO: Container volume-tester ready: false, restart count 0 Mar 27 23:54:55.265: INFO: up-down-2-w7vfg started at 2023-03-27 23:54:19 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:55.265: INFO: Container up-down-2 ready: true, restart count 0 Mar 27 23:54:55.265: INFO: pfpod started at 2023-03-27 23:52:51 +0000 UTC (0+2 container statuses recorded) Mar 27 23:54:55.265: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:54:55.265: INFO: Container readiness ready: false, restart count 0 Mar 27 23:54:55.265: INFO: kube-proxy-i-0b266cca2f40af1c1 started at 2023-03-27 23:29:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:55.265: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:54:55.265: INFO: calico-node-nmfrn started at 2023-03-27 23:29:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:54:55.265: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:54:55.265: INFO: Init container install-cni ready: true, restart count 1 Mar 27 23:54:55.265: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:54:55.265: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:54:55.265: INFO: inline-volume-tester-8gmqj started at 2023-03-27 23:54:46 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:55.265: INFO: Container csi-volume-tester ready: true, restart count 0 E0327 23:54:55.793909 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:54:56.011: INFO: Latency metrics for node i-0b266cca2f40af1c1 Mar 27 23:54:56.011: INFO: Logging node info for node i-0d2b1184f62de758a Mar 27 23:54:56.189: INFO: Node Info: &Node{ObjectMeta:{i-0d2b1184f62de758a 2e42ccb1-9580-4126-b5d6-05a5f98403c5 45677 0 2023-03-27 23:17:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d2b1184f62de758a kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d2b1184f62de758a"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.6/19 projectcalico.org/IPv4IPIPTunnelAddr:100.120.150.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-27 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-27 23:17:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:18:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:50:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0d2b1184f62de758a,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:18:49 +0000 UTC,LastTransitionTime:2023-03-27 23:18:49 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:18:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.6,},NodeAddress{Type:ExternalIP,Address:3.35.11.126,},NodeAddress{Type:InternalDNS,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-35-11-126.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2781baf4898d15db3c853bf5b6d100,SystemUUID:ec2781ba-f489-8d15-db3c-853bf5b6d100,BootID:5e905df0-494e-4050-b6d7-400763c3dd66,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45208814,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43629012,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074727,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:54:56.189: INFO: Logging kubelet events for node i-0d2b1184f62de758a Mar 27 23:54:56.374: INFO: Logging pods the kubelet thinks is on node i-0d2b1184f62de758a Mar 27 23:54:56.557: INFO: ebs-csi-controller-79f984545f-42vd6 started at 2023-03-27 23:19:48 +0000 UTC (0+5 container statuses recorded) Mar 27 23:54:56.557: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:54:56.557: INFO: etcd-manager-events-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:54:56.557: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:54:56.557: INFO: etcd-manager-main-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:54:56.557: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:54:56.557: INFO: kube-apiserver-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+2 container statuses recorded) Mar 27 23:54:56.557: INFO: Container healthcheck ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Container kube-apiserver ready: true, restart count 3 Mar 27 23:54:56.557: INFO: kube-controller-manager-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:56.557: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 27 23:54:56.557: INFO: kube-proxy-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:56.557: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:54:56.557: INFO: aws-cloud-controller-manager-776ld started at 2023-03-27 23:18:28 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:56.557: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 27 23:54:56.557: INFO: calico-kube-controllers-55f488656-vj6gh started at 2023-03-27 23:18:57 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:56.557: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 27 23:54:56.557: INFO: kube-scheduler-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:56.557: INFO: Container kube-scheduler ready: true, restart count 0 Mar 27 23:54:56.557: INFO: dns-controller-69987775c6-rzzjj started at 2023-03-27 23:18:25 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:56.557: INFO: Container dns-controller ready: true, restart count 0 Mar 27 23:54:56.557: INFO: calico-node-scbv2 started at 2023-03-27 23:18:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:54:56.557: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:54:56.557: INFO: ebs-csi-node-nvqcx started at 2023-03-27 23:18:29 +0000 UTC (0+3 container statuses recorded) Mar 27 23:54:56.557: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:54:56.557: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:54:56.557: INFO: kops-controller-w4b7t started at 2023-03-27 23:18:29 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:56.557: INFO: Container kops-controller ready: true, restart count 0 E0327 23:54:56.794175 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:54:57.150: INFO: Latency metrics for node i-0d2b1184f62de758a Mar 27 23:54:57.150: INFO: Logging node info for node i-0e1f2cd7cf96d3020 Mar 27 23:54:57.329: INFO: Node Info: &Node{ObjectMeta:{i-0e1f2cd7cf96d3020 efcf1e66-db0e-4256-a849-b9c377eef04e 56879 0 2023-03-27 23:22:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e1f2cd7cf96d3020 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0e1f2cd7cf96d3020 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-provisioning-35":"i-0e1f2cd7cf96d3020","ebs.csi.aws.com":"i-0e1f2cd7cf96d3020"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.120/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.86.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:22:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:22:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:54:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0e1f2cd7cf96d3020,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:22:52 +0000 UTC,LastTransitionTime:2023-03-27 23:22:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:35 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:35 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:35 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:54:35 +0000 UTC,LastTransitionTime:2023-03-27 23:22:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.120,},NodeAddress{Type:ExternalIP,Address:15.164.224.157,},NodeAddress{Type:InternalDNS,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-164-224-157.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29484c87c97fb3d8a52788d5971660,SystemUUID:ec29484c-87c9-7fb3-d8a5-2788d5971660,BootID:b73c540f-bd49-470a-a7b3-08c31cc8c2fd,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:54:57.329: INFO: Logging kubelet events for node i-0e1f2cd7cf96d3020 Mar 27 23:54:57.518: INFO: Logging pods the kubelet thinks is on node i-0e1f2cd7cf96d3020 Mar 27 23:54:57.701: INFO: coredns-559769c974-scn7m started at 2023-03-27 23:30:58 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:57.701: INFO: Container coredns ready: true, restart count 0 Mar 27 23:54:57.701: INFO: hostexec-i-0e1f2cd7cf96d3020-sj6dd started at 2023-03-27 23:54:32 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:57.701: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:54:57.701: INFO: local-client started at 2023-03-27 23:54:56 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:57.701: INFO: Container local-client ready: false, restart count 0 Mar 27 23:54:57.701: INFO: csi-hostpathplugin-0 started at 2023-03-27 23:54:16 +0000 UTC (0+7 container statuses recorded) Mar 27 23:54:57.701: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:54:57.701: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:54:57.701: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:54:57.701: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 27 23:54:57.701: INFO: Container hostpath ready: true, restart count 0 Mar 27 23:54:57.701: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:54:57.701: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:54:57.701: INFO: ebs-csi-node-vcn4h started at 2023-03-27 23:22:28 +0000 UTC (0+3 container statuses recorded) Mar 27 23:54:57.701: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:54:57.701: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:54:57.701: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:54:57.701: INFO: netserver-3 started at 2023-03-27 23:54:44 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:57.701: INFO: Container webserver ready: true, restart count 0 Mar 27 23:54:57.701: INFO: kube-proxy-i-0e1f2cd7cf96d3020 started at 2023-03-27 23:22:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:57.701: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:54:57.701: INFO: calico-node-9fzr7 started at 2023-03-27 23:22:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:54:57.701: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:54:57.701: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:54:57.701: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:54:57.701: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:54:57.701: INFO: coredns-autoscaler-7cb5c5b969-2k8xf started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:54:57.701: INFO: Container autoscaler ready: true, restart count 0 E0327 23:54:57.794510 7310 retrywatcher.go:130] "Watch failed" err="context canceled" Mar 27 23:54:58.385: INFO: Latency metrics for node i-0e1f2cd7cf96d3020 END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:54:58.385 (6.552s) < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - dump namespaces | framework.go:209 @ 03/27/23 23:54:58.385 (6.552s) > Enter [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - tear down framework | framework.go:206 @ 03/27/23 23:54:58.385 STEP: Destroying namespace "volume-8869" for this suite. - test/e2e/framework/framework.go:351 @ 03/27/23 23:54:58.385 < Exit [DeferCleanup (Each)] [Testpattern: Dynamic PV (ext4)] volumes - tear down framework | framework.go:206 @ 03/27/23 23:54:58.564 (180ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:54:58.565 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:54:58.565 (0s)
Filter through log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\sExternal\sStorage\s\[Driver\:\sebs\.csi\.aws\.com\]\s\[Testpattern\:\sGeneric\sEphemeral\-volume\s\(default\sfs\)\s\(late\-binding\)\]\sephemeral\sshould\ssupport\stwo\spods\swhich\shave\sthe\ssame\svolume\sdefinition$'
[FAILED] expected pod to not be found: Timed out after 300.000s. Expected <*v1.Pod | 0xc0028da000>: metadata: annotations: cni.projectcalico.org/containerID: 601557257994b52e63a5fc194cdd7608bb7b4714683437d4f4ad03a9d2fecb15 cni.projectcalico.org/podIP: "" cni.projectcalico.org/podIPs: "" creationTimestamp: "2023-03-27T23:43:53Z" deletionGracePeriodSeconds: 30 deletionTimestamp: "2023-03-27T23:45:41Z" finalizers: - foregroundDeletion generateName: inline-volume-tester- labels: app: inline-volume-tester managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:app: {} f:spec: f:containers: k:{"name":"csi-volume-tester"}: .: {} f:command: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/mnt/test-0"}: .: {} f:mountPath: {} f:name: {} f:dnsPolicy: {} f:enableServiceLinks: {} f:nodeSelector: {} f:restartPolicy: {} f:schedulerName: {} f:securityContext: {} f:terminationGracePeriodSeconds: {} f:volumes: .: {} k:{"name":"my-volume-0"}: .: {} f:ephemeral: .: {} f:volumeClaimTemplate: .: {} f:metadata: .: {} f:creationTimestamp: {} f:spec: .: {} f:accessModes: {} f:resources: .: {} f:requests: .: {} f:storage: {} f:storageClassName: {} f:volumeMode: {} f:name: {} manager: e2e.test operation: Update time: "2023-03-27T23:43:53Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: .: {} k:{"type":"PodScheduled"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} manager: kube-scheduler operation: Update subresource: status time: "2023-03-27T23:43:53Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:cni.projectcalico.org/containerID: {} f:cni.projectcalico.org/podIP: {} f:cni.projectcalico.org/podIPs: {} manager: calico operation: Update subresource: status time: "2023-03-27T23:45:43Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: k:{"type":"ContainersReady"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:reason: {} f:status: {} f:type: {} k:{"type":"Initialized"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:status: {} f:type: {} k:{"type":"Ready"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:reason: {} f:status: {} f:type: {} f:containerStatuses: {} f:hostIP: {} f:phase: {} f:podIP: {} f:podIPs: .: {} k:{"ip":"100.113.211.153"}: .: {} f:ip: {} f:startTime: {} manager: kubelet operation: Update subresource: status time: "2023-03-27T23:45:44Z" name: inline-volume-tester-kwwd2 namespace: ephemeral-7265 resourceVersion: "31442" uid: 7b1b502d-7c87-442d-a498-ae7aaf4fd286 spec: containers: - command: - /bin/sh - -c - sleep 10000 image: registry.k8s.io/e2e-test-images/busybox:1.29-4 imagePullPolicy: IfNotPresent name: csi-volume-tester resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /mnt/test-0 name: my-volume-0 - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-929nx readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: i-05ec0d9f3272223ec nodeSelector: kubernetes.io/os: linux preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Never schedulerName: default-scheduler securityContext: {} serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - ephemeral: volumeClaimTemplate: metadata: creationTimestamp: null spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: ephemeral-7265-e2e-scs8tw5 volumeMode: Filesystem name: my-volume-0 - name: kube-api-access-929nx projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions: - lastProbeTime: null lastTransitionTime: "2023-03-27T23:43:59Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2023-03-27T23:45:44Z" reason: PodFailed status: "False" type: Ready - lastProbeTime: null lastTransitionTime: "2023-03-27T23:45:44Z" reason: PodFailed status: "False" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2023-03-27T23:43:59Z" status: "True" type: PodScheduled containerStatuses: - containerID: containerd://bd73f8eef1c7a013792e30eea1b2e058dfb8a33fabebf12a9c5b8dc63bb731d2 image: registry.k8s.io/e2e-test-images/busybox:1.29-4 imageID: registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 lastState: {} name: csi-volume-tester ready: false restartCount: 0 started: false state: terminated: containerID: containerd://bd73f8eef1c7a013792e30eea1b2e058dfb8a33fabebf12a9c5b8dc63bb731d2 exitCode: 137 finishedAt: "2023-03-27T23:45:41Z" reason: Error startedAt: "2023-03-27T23:44:03Z" hostIP: 172.20.52.36 phase: Failed podIP: 100.113.211.153 podIPs: - ip: 100.113.211.153 qosClass: BestEffort startTime: "2023-03-27T23:43:59Z" to be nil In [It] at: test/e2e/storage/testsuites/provisioning.go:1097 @ 03/27/23 23:55:11.885from junit_01.xml
> Enter [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:43:49.167 < Exit [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/storage/framework/testsuite.go:51 @ 03/27/23 23:43:49.167 (0s) > Enter [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - set up framework | framework.go:191 @ 03/27/23 23:43:49.167 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/27/23 23:43:49.167 Mar 27 23:43:49.167: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename ephemeral - test/e2e/framework/framework.go:250 @ 03/27/23 23:43:49.169 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/27/23 23:43:49.705 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/27/23 23:43:50.057 < Exit [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - set up framework | framework.go:191 @ 03/27/23 23:43:50.411 (1.244s) > Enter [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:43:50.411 < Exit [BeforeEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:43:50.411 (0s) > Enter [It] should support two pods which have the same volume definition - test/e2e/storage/testsuites/ephemeral.go:284 @ 03/27/23 23:43:50.411 Mar 27 23:43:50.773: INFO: Error getting logs for pod inline-volume-r269g: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/ephemeral-7265/pods/inline-volume-r269g/log": stream error: stream ID 1259; INTERNAL_ERROR; received from peer Mar 27 23:43:50.950: INFO: Deleting pod "inline-volume-r269g" in namespace "ephemeral-7265" Mar 27 23:43:51.128: INFO: Wait up to 5m0s for pod "inline-volume-r269g" to be fully deleted Mar 27 23:43:53.516: INFO: Creating resource for dynamic PV Mar 27 23:43:53.516: INFO: Using claimSize:1Gi, test suite supported size:{ }, driver(ebs.csi.aws.com) supported size:{ } STEP: creating a StorageClass ephemeral-7265-e2e-scs8tw5 - test/e2e/storage/framework/volume_resource.go:102 @ 03/27/23 23:43:53.517 STEP: checking the requested inline volume exists in the pod running on node {Name: Selector:map[kubernetes.io/os:linux] Affinity:nil} - test/e2e/storage/testsuites/ephemeral.go:389 @ 03/27/23 23:43:53.696 STEP: writing data in one pod and checking the second does not see it (it should get its own volume) - test/e2e/storage/testsuites/ephemeral.go:310 @ 03/27/23 23:44:18.566 Mar 27 23:44:18.566: INFO: ExecWithOptions {Command:[/bin/sh -c touch /mnt/test-0/hello-world] Namespace:ephemeral-7265 PodName:inline-volume-tester-kwwd2 ContainerName:csi-volume-tester Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false} Mar 27 23:44:18.566: INFO: >>> kubeConfig: /root/.kube/config Mar 27 23:44:18.566: INFO: ExecWithOptions: Clientset creation Mar 27 23:44:18.567: INFO: ExecWithOptions: execute(POST https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/ephemeral-7265/pods/inline-volume-tester-kwwd2/exec?command=%2Fbin%2Fsh&command=-c&command=touch+%2Fmnt%2Ftest-0%2Fhello-world&container=csi-volume-tester&container=csi-volume-tester&stderr=true&stdout=true) Mar 27 23:44:19.826: INFO: ExecWithOptions {Command:[/bin/sh -c [ ! -f /mnt/test-0/hello-world ]] Namespace:ephemeral-7265 PodName:inline-volume-tester2-wpxz6 ContainerName:csi-volume-tester Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false} Mar 27 23:44:19.826: INFO: >>> kubeConfig: /root/.kube/config Mar 27 23:44:19.827: INFO: ExecWithOptions: Clientset creation Mar 27 23:44:19.827: INFO: ExecWithOptions: execute(POST https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/ephemeral-7265/pods/inline-volume-tester2-wpxz6/exec?command=%2Fbin%2Fsh&command=-c&command=%5B+%21+-f+%2Fmnt%2Ftest-0%2Fhello-world+%5D&container=csi-volume-tester&container=csi-volume-tester&stderr=true&stdout=true) Mar 27 23:44:21.171: INFO: Error getting logs for pod inline-volume-tester2-wpxz6: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/ephemeral-7265/pods/inline-volume-tester2-wpxz6/log": stream error: stream ID 1303; INTERNAL_ERROR; received from peer Mar 27 23:44:21.703: INFO: Deleting pod "inline-volume-tester2-wpxz6" in namespace "ephemeral-7265" Mar 27 23:44:21.890: INFO: Wait up to 5m0s for pod "inline-volume-tester2-wpxz6" to be fully deleted Mar 27 23:44:54.731: INFO: Wait up to 5m0s for pod PV pvc-7ef43b49-4a0b-4226-a220-39cecfcf9ae2 to be fully deleted Mar 27 23:44:54.731: INFO: Waiting up to 5m0s for PersistentVolume pvc-7ef43b49-4a0b-4226-a220-39cecfcf9ae2 to get deleted Mar 27 23:44:54.916: INFO: PersistentVolume pvc-7ef43b49-4a0b-4226-a220-39cecfcf9ae2 found and phase=Released (184.248919ms) Mar 27 23:45:00.093: INFO: PersistentVolume pvc-7ef43b49-4a0b-4226-a220-39cecfcf9ae2 found and phase=Released (5.361835748s) Mar 27 23:45:05.270: INFO: PersistentVolume pvc-7ef43b49-4a0b-4226-a220-39cecfcf9ae2 found and phase=Released (10.53891224s) Mar 27 23:45:10.448: INFO: PersistentVolume pvc-7ef43b49-4a0b-4226-a220-39cecfcf9ae2 was removed Mar 27 23:45:10.628: INFO: Error getting logs for pod inline-volume-tester-kwwd2: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/ephemeral-7265/pods/inline-volume-tester-kwwd2/log": stream error: stream ID 1353; INTERNAL_ERROR; received from peer Mar 27 23:45:10.991: INFO: Deleting pod "inline-volume-tester-kwwd2" in namespace "ephemeral-7265" Mar 27 23:45:11.171: INFO: Wait up to 5m0s for pod "inline-volume-tester-kwwd2" to be fully deleted Mar 27 23:50:11.174: INFO: Failed inside E2E framework: k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodNotFoundInNamespace({0x7f50480b58d8, 0xc001f67bf0}, {0x72b26d0?, 0xc0042deea0}, {0xc001aab600, 0x1a}, {0xc001c16710, 0xe}, 0x0?) test/e2e/framework/pod/wait.go:538 +0x190 k8s.io/kubernetes/test/e2e/storage/testsuites.StopPodAndDependents({0x7f50480b58d8, 0xc001f67bf0}, {0x72b26d0?, 0xc0042deea0?}, 0xc0014640a0, 0xc00210c000) test/e2e/storage/testsuites/provisioning.go:1097 +0xa8e k8s.io/kubernetes/test/e2e/storage/testsuites.EphemeralTest.TestEphemeral({{0x72b26d0, 0xc0042deea0}, 0xc0014640a0, {0xc002f2e590, 0xe}, {0x0, 0x0}, 0xc004aba0f0, {{0x0, 0x0}, ...}, ...}, ...) test/e2e/storage/testsuites/ephemeral.go:434 +0x7aa k8s.io/kubernetes/test/e2e/storage/testsuites.(*ephemeralTestSuite).DefineTests.func6({0x7f50480b58d8, 0xc001f67bf0}) test/e2e/storage/testsuites/ephemeral.go:323 +0x234 Mar 27 23:50:11.351: INFO: Error getting logs for pod inline-volume-tester-kwwd2: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/ephemeral-7265/pods/inline-volume-tester-kwwd2/log": stream error: stream ID 1637; INTERNAL_ERROR; received from peer Mar 27 23:50:11.704: INFO: Deleting pod "inline-volume-tester-kwwd2" in namespace "ephemeral-7265" Mar 27 23:50:11.881: INFO: Wait up to 5m0s for pod "inline-volume-tester-kwwd2" to be fully deleted Mar 27 23:55:11.884: INFO: Failed inside E2E framework: k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodNotFoundInNamespace({0x7f50480b58d8, 0xc001f67bf0}, {0x72b26d0?, 0xc0042deea0}, {0xc001aab600, 0x1a}, {0xc001c16710, 0xe}, 0x0?) test/e2e/framework/pod/wait.go:538 +0x190 k8s.io/kubernetes/test/e2e/storage/testsuites.StopPodAndDependents({0x7f50480b58d8, 0xc001f67bf0}, {0x72b26d0?, 0xc0042deea0?}, 0xc0014640a0, 0xc00210c000) test/e2e/storage/testsuites/provisioning.go:1097 +0xa8e k8s.io/kubernetes/test/e2e/storage/testsuites.EphemeralTest.TestEphemeral.func1() test/e2e/storage/testsuites/ephemeral.go:421 +0x38 panic({0x66383e0, 0xc00078a7e0}) /usr/local/go/src/runtime/panic.go:884 +0x213 k8s.io/kubernetes/test/e2e/framework.ExpectNoErrorWithOffset(0x1, {0x7240b00, 0xc000c77360}, {0x0?, 0xc001aab600?, 0x1a?}) test/e2e/framework/expect.go:363 +0x363 k8s.io/kubernetes/test/e2e/framework.ExpectNoError(...) test/e2e/framework/expect.go:320 k8s.io/kubernetes/test/e2e/storage/testsuites.StopPodAndDependents({0x7f50480b58d8, 0xc001f67bf0}, {0x72b26d0?, 0xc0042deea0?}, 0xc0014640a0, 0xc00210c000) test/e2e/storage/testsuites/provisioning.go:1097 +0xaa5 k8s.io/kubernetes/test/e2e/storage/testsuites.EphemeralTest.TestEphemeral({{0x72b26d0, 0xc0042deea0}, 0xc0014640a0, {0xc002f2e590, 0xe}, {0x0, 0x0}, 0xc004aba0f0, {{0x0, 0x0}, ...}, ...}, ...) test/e2e/storage/testsuites/ephemeral.go:434 +0x7aa k8s.io/kubernetes/test/e2e/storage/testsuites.(*ephemeralTestSuite).DefineTests.func6({0x7f50480b58d8, 0xc001f67bf0}) test/e2e/storage/testsuites/ephemeral.go:323 +0x234 [FAILED] expected pod to not be found: Timed out after 300.000s. Expected <*v1.Pod | 0xc0028da000>: metadata: annotations: cni.projectcalico.org/containerID: 601557257994b52e63a5fc194cdd7608bb7b4714683437d4f4ad03a9d2fecb15 cni.projectcalico.org/podIP: "" cni.projectcalico.org/podIPs: "" creationTimestamp: "2023-03-27T23:43:53Z" deletionGracePeriodSeconds: 30 deletionTimestamp: "2023-03-27T23:45:41Z" finalizers: - foregroundDeletion generateName: inline-volume-tester- labels: app: inline-volume-tester managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:app: {} f:spec: f:containers: k:{"name":"csi-volume-tester"}: .: {} f:command: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/mnt/test-0"}: .: {} f:mountPath: {} f:name: {} f:dnsPolicy: {} f:enableServiceLinks: {} f:nodeSelector: {} f:restartPolicy: {} f:schedulerName: {} f:securityContext: {} f:terminationGracePeriodSeconds: {} f:volumes: .: {} k:{"name":"my-volume-0"}: .: {} f:ephemeral: .: {} f:volumeClaimTemplate: .: {} f:metadata: .: {} f:creationTimestamp: {} f:spec: .: {} f:accessModes: {} f:resources: .: {} f:requests: .: {} f:storage: {} f:storageClassName: {} f:volumeMode: {} f:name: {} manager: e2e.test operation: Update time: "2023-03-27T23:43:53Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: .: {} k:{"type":"PodScheduled"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} manager: kube-scheduler operation: Update subresource: status time: "2023-03-27T23:43:53Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:cni.projectcalico.org/containerID: {} f:cni.projectcalico.org/podIP: {} f:cni.projectcalico.org/podIPs: {} manager: calico operation: Update subresource: status time: "2023-03-27T23:45:43Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: k:{"type":"ContainersReady"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:reason: {} f:status: {} f:type: {} k:{"type":"Initialized"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:status: {} f:type: {} k:{"type":"Ready"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:reason: {} f:status: {} f:type: {} f:containerStatuses: {} f:hostIP: {} f:phase: {} f:podIP: {} f:podIPs: .: {} k:{"ip":"100.113.211.153"}: .: {} f:ip: {} f:startTime: {} manager: kubelet operation: Update subresource: status time: "2023-03-27T23:45:44Z" name: inline-volume-tester-kwwd2 namespace: ephemeral-7265 resourceVersion: "31442" uid: 7b1b502d-7c87-442d-a498-ae7aaf4fd286 spec: containers: - command: - /bin/sh - -c - sleep 10000 image: registry.k8s.io/e2e-test-images/busybox:1.29-4 imagePullPolicy: IfNotPresent name: csi-volume-tester resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /mnt/test-0 name: my-volume-0 - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-929nx readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: i-05ec0d9f3272223ec nodeSelector: kubernetes.io/os: linux preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Never schedulerName: default-scheduler securityContext: {} serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - ephemeral: volumeClaimTemplate: metadata: creationTimestamp: null spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: ephemeral-7265-e2e-scs8tw5 volumeMode: Filesystem name: my-volume-0 - name: kube-api-access-929nx projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions: - lastProbeTime: null lastTransitionTime: "2023-03-27T23:43:59Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2023-03-27T23:45:44Z" reason: PodFailed status: "False" type: Ready - lastProbeTime: null lastTransitionTime: "2023-03-27T23:45:44Z" reason: PodFailed status: "False" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2023-03-27T23:43:59Z" status: "True" type: PodScheduled containerStatuses: - containerID: containerd://bd73f8eef1c7a013792e30eea1b2e058dfb8a33fabebf12a9c5b8dc63bb731d2 image: registry.k8s.io/e2e-test-images/busybox:1.29-4 imageID: registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 lastState: {} name: csi-volume-tester ready: false restartCount: 0 started: false state: terminated: containerID: containerd://bd73f8eef1c7a013792e30eea1b2e058dfb8a33fabebf12a9c5b8dc63bb731d2 exitCode: 137 finishedAt: "2023-03-27T23:45:41Z" reason: Error startedAt: "2023-03-27T23:44:03Z" hostIP: 172.20.52.36 phase: Failed podIP: 100.113.211.153 podIPs: - ip: 100.113.211.153 qosClass: BestEffort startTime: "2023-03-27T23:43:59Z" to be nil In [It] at: test/e2e/storage/testsuites/provisioning.go:1097 @ 03/27/23 23:55:11.885 < Exit [It] should support two pods which have the same volume definition - test/e2e/storage/testsuites/ephemeral.go:284 @ 03/27/23 23:55:11.885 (11m21.474s) > Enter [AfterEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:55:11.885 Mar 27 23:55:11.885: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:55:12.064 (180ms) > Enter [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/storage/testsuites/ephemeral.go:286 @ 03/27/23 23:55:12.064 STEP: Deleting sc - test/e2e/storage/framework/volume_resource.go:228 @ 03/27/23 23:55:12.064 < Exit [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/storage/testsuites/ephemeral.go:286 @ 03/27/23 23:55:12.417 (353ms) > Enter [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:55:12.417 < Exit [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:55:12.417 (0s) > Enter [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - dump namespaces | framework.go:209 @ 03/27/23 23:55:12.417 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:55:12.417 STEP: Collecting events from namespace "ephemeral-7265". - test/e2e/framework/debug/dump.go:42 @ 03/27/23 23:55:12.417 STEP: Found 23 events. - test/e2e/framework/debug/dump.go:46 @ 03/27/23 23:55:12.595 Mar 27 23:55:12.595: INFO: At 2023-03-27 23:43:50 +0000 UTC - event for inline-volume-r269g: {default-scheduler } FailedScheduling: 0/5 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "inline-volume-r269g-my-volume". preemption: 0/5 nodes are available: 5 No preemption victims found for incoming pod.. Mar 27 23:55:12.595: INFO: At 2023-03-27 23:43:50 +0000 UTC - event for inline-volume-r269g-my-volume: {persistentvolume-controller } ProvisioningFailed: storageclass.storage.k8s.io "no-such-storage-class" not found Mar 27 23:55:12.595: INFO: At 2023-03-27 23:43:53 +0000 UTC - event for inline-volume-tester-kwwd2: {default-scheduler } FailedScheduling: 0/5 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "inline-volume-tester-kwwd2-my-volume-0". preemption: 0/5 nodes are available: 5 No preemption victims found for incoming pod.. Mar 27 23:55:12.595: INFO: At 2023-03-27 23:43:53 +0000 UTC - event for inline-volume-tester-kwwd2-my-volume-0: {persistentvolume-controller } WaitForPodScheduled: waiting for pod inline-volume-tester-kwwd2 to be scheduled Mar 27 23:55:12.595: INFO: At 2023-03-27 23:43:55 +0000 UTC - event for inline-volume-tester-kwwd2-my-volume-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 27 23:55:12.595: INFO: At 2023-03-27 23:43:55 +0000 UTC - event for inline-volume-tester-kwwd2-my-volume-0: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } Provisioning: External provisioner is provisioning volume for claim "ephemeral-7265/inline-volume-tester-kwwd2-my-volume-0" Mar 27 23:55:12.595: INFO: At 2023-03-27 23:43:58 +0000 UTC - event for inline-volume-tester-kwwd2-my-volume-0: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } ProvisioningSucceeded: Successfully provisioned volume pvc-32f66270-d7b0-47b7-b64f-4c168e09e6f1 Mar 27 23:55:12.595: INFO: At 2023-03-27 23:43:59 +0000 UTC - event for inline-volume-tester-kwwd2: {default-scheduler } Scheduled: Successfully assigned ephemeral-7265/inline-volume-tester-kwwd2 to i-05ec0d9f3272223ec Mar 27 23:55:12.595: INFO: At 2023-03-27 23:44:01 +0000 UTC - event for inline-volume-tester-kwwd2: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-32f66270-d7b0-47b7-b64f-4c168e09e6f1" Mar 27 23:55:12.595: INFO: At 2023-03-27 23:44:03 +0000 UTC - event for inline-volume-tester-kwwd2: {kubelet i-05ec0d9f3272223ec} Created: Created container csi-volume-tester Mar 27 23:55:12.595: INFO: At 2023-03-27 23:44:03 +0000 UTC - event for inline-volume-tester-kwwd2: {kubelet i-05ec0d9f3272223ec} Started: Started container csi-volume-tester Mar 27 23:55:12.595: INFO: At 2023-03-27 23:44:03 +0000 UTC - event for inline-volume-tester-kwwd2: {kubelet i-05ec0d9f3272223ec} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 27 23:55:12.595: INFO: At 2023-03-27 23:44:05 +0000 UTC - event for inline-volume-tester2-wpxz6-my-volume-0: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } Provisioning: External provisioner is provisioning volume for claim "ephemeral-7265/inline-volume-tester2-wpxz6-my-volume-0" Mar 27 23:55:12.595: INFO: At 2023-03-27 23:44:05 +0000 UTC - event for inline-volume-tester2-wpxz6-my-volume-0: {persistentvolume-controller } WaitForPodScheduled: waiting for pod inline-volume-tester2-wpxz6 to be scheduled Mar 27 23:55:12.595: INFO: At 2023-03-27 23:44:05 +0000 UTC - event for inline-volume-tester2-wpxz6-my-volume-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Mar 27 23:55:12.595: INFO: At 2023-03-27 23:44:08 +0000 UTC - event for inline-volume-tester2-wpxz6-my-volume-0: {ebs.csi.aws.com_i-0d2b1184f62de758a_7b6d7817-2259-4f33-b441-9055a33f2a5f } ProvisioningSucceeded: Successfully provisioned volume pvc-7ef43b49-4a0b-4226-a220-39cecfcf9ae2 Mar 27 23:55:12.595: INFO: At 2023-03-27 23:44:09 +0000 UTC - event for inline-volume-tester2-wpxz6: {default-scheduler } Scheduled: Successfully assigned ephemeral-7265/inline-volume-tester2-wpxz6 to i-0b266cca2f40af1c1 Mar 27 23:55:12.595: INFO: At 2023-03-27 23:44:11 +0000 UTC - event for inline-volume-tester2-wpxz6: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-7ef43b49-4a0b-4226-a220-39cecfcf9ae2" Mar 27 23:55:12.595: INFO: At 2023-03-27 23:44:15 +0000 UTC - event for inline-volume-tester2-wpxz6: {kubelet i-0b266cca2f40af1c1} Pulled: Container image "registry.k8s.io/e2e-test-images/busybox:1.29-4" already present on machine Mar 27 23:55:12.595: INFO: At 2023-03-27 23:44:15 +0000 UTC - event for inline-volume-tester2-wpxz6: {kubelet i-0b266cca2f40af1c1} Created: Created container csi-volume-tester Mar 27 23:55:12.595: INFO: At 2023-03-27 23:44:15 +0000 UTC - event for inline-volume-tester2-wpxz6: {kubelet i-0b266cca2f40af1c1} Started: Started container csi-volume-tester Mar 27 23:55:12.595: INFO: At 2023-03-27 23:44:21 +0000 UTC - event for inline-volume-tester2-wpxz6: {kubelet i-0b266cca2f40af1c1} Killing: Stopping container csi-volume-tester Mar 27 23:55:12.595: INFO: At 2023-03-27 23:45:11 +0000 UTC - event for inline-volume-tester-kwwd2: {kubelet i-05ec0d9f3272223ec} Killing: Stopping container csi-volume-tester Mar 27 23:55:12.771: INFO: POD NODE PHASE GRACE CONDITIONS Mar 27 23:55:12.771: INFO: inline-volume-tester-kwwd2 i-05ec0d9f3272223ec Failed 30s [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-27 23:43:59 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-27 23:45:44 +0000 UTC PodFailed } {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-03-27 23:45:44 +0000 UTC PodFailed } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-27 23:43:59 +0000 UTC }] Mar 27 23:55:12.771: INFO: Mar 27 23:55:12.950: INFO: Unable to fetch ephemeral-7265/inline-volume-tester-kwwd2/csi-volume-tester logs: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/ephemeral-7265/pods/inline-volume-tester-kwwd2/log?container=csi-volume-tester&previous=false&tailLines=42": stream error: stream ID 1929; INTERNAL_ERROR; received from peer Mar 27 23:55:13.128: INFO: Logging node info for node i-022003b3a45876d5d Mar 27 23:55:13.304: INFO: Node Info: &Node{ObjectMeta:{i-022003b3a45876d5d 20348b58-b610-41a0-accb-ca195a540336 56891 0 2023-03-27 23:33:26 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-022003b3a45876d5d kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-022003b3a45876d5d topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-022003b3a45876d5d"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.49.118/19 projectcalico.org/IPv4IPIPTunnelAddr:100.115.192.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:33:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:33:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:54:48 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:54:50 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-022003b3a45876d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:33:51 +0000 UTC,LastTransitionTime:2023-03-27 23:33:51 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:50 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:50 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:50 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:54:50 +0000 UTC,LastTransitionTime:2023-03-27 23:33:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.118,},NodeAddress{Type:ExternalIP,Address:54.180.103.89,},NodeAddress{Type:InternalDNS,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-103-89.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccb51411bbbb92946da93b9a452fe,SystemUUID:ec2ccb51-411b-bbb9-2946-da93b9a452fe,BootID:841d91a9-bfa5-4e0f-9279-7c83efeef8ca,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-027c7267daa9fbf92],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-027c7267daa9fbf92,DevicePath:,},},Config:nil,},} Mar 27 23:55:13.304: INFO: Logging kubelet events for node i-022003b3a45876d5d Mar 27 23:55:13.489: INFO: Logging pods the kubelet thinks is on node i-022003b3a45876d5d Mar 27 23:55:13.673: INFO: up-down-2-jwgr8 started at 2023-03-27 23:54:19 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.673: INFO: Container up-down-2 ready: true, restart count 0 Mar 27 23:55:13.673: INFO: verify-service-up-exec-pod-s56qk started at 2023-03-27 23:55:08 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.673: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:13.673: INFO: up-down-3-k947m started at 2023-03-27 23:55:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.673: INFO: Container up-down-3 ready: true, restart count 0 Mar 27 23:55:13.673: INFO: pod2 started at 2023-03-27 23:55:07 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.673: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:13.673: INFO: calico-node-2wjdb started at 2023-03-27 23:33:27 +0000 UTC (3+1 container statuses recorded) Mar 27 23:55:13.673: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:55:13.673: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:55:13.673: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:55:13.673: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:55:13.673: INFO: webserver-pod started at 2023-03-27 23:52:55 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.673: INFO: Container agnhost ready: false, restart count 0 Mar 27 23:55:13.673: INFO: hostexec-i-022003b3a45876d5d-6w5x9 started at 2023-03-27 23:55:12 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.673: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:13.673: INFO: hostexec-i-022003b3a45876d5d-4p4t4 started at 2023-03-27 23:55:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.673: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:13.673: INFO: netserver-0 started at 2023-03-27 23:54:44 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.673: INFO: Container webserver ready: true, restart count 0 Mar 27 23:55:13.673: INFO: kube-proxy-i-022003b3a45876d5d started at 2023-03-27 23:33:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.673: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:55:13.673: INFO: ebs-csi-node-2knd4 started at 2023-03-27 23:33:27 +0000 UTC (0+3 container statuses recorded) Mar 27 23:55:13.673: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:55:13.673: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:55:13.673: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:55:13.673: INFO: pfpod started at 2023-03-27 23:52:53 +0000 UTC (0+2 container statuses recorded) Mar 27 23:55:13.673: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:55:13.673: INFO: Container readiness ready: false, restart count 0 Mar 27 23:55:13.673: INFO: pfpod started at 2023-03-27 23:53:13 +0000 UTC (0+2 container statuses recorded) Mar 27 23:55:13.673: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:55:13.673: INFO: Container readiness ready: false, restart count 0 Mar 27 23:55:13.673: INFO: ss-0 started at 2023-03-27 23:54:46 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.673: INFO: Container webserver ready: true, restart count 0 Mar 27 23:55:13.673: INFO: hostexec-i-022003b3a45876d5d-wp6rd started at 2023-03-27 23:55:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:13.673: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:14.364: INFO: Latency metrics for node i-022003b3a45876d5d Mar 27 23:55:14.364: INFO: Logging node info for node i-05ec0d9f3272223ec Mar 27 23:55:14.541: INFO: Node Info: &Node{ObjectMeta:{i-05ec0d9f3272223ec 365deda4-148e-4593-abdf-3103630fef28 57431 0 2023-03-27 23:25:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c io.kubernetes.storage.mock/node:some-mock-node kubernetes.io/arch:amd64 kubernetes.io/hostname:i-05ec0d9f3272223ec kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-05ec0d9f3272223ec topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-05ec0d9f3272223ec"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.52.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.113.211.128 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:25:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:25:34 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.7.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:25:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:54:59 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:55:05 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:io.kubernetes.storage.mock/node":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.7.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-05ec0d9f3272223ec,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.7.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:25:42 +0000 UTC,LastTransitionTime:2023-03-27 23:25:42 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:55:05 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:55:05 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:55:05 +0000 UTC,LastTransitionTime:2023-03-27 23:25:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:55:05 +0000 UTC,LastTransitionTime:2023-03-27 23:25:34 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.52.36,},NodeAddress{Type:ExternalIP,Address:3.34.125.198,},NodeAddress{Type:InternalDNS,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-05ec0d9f3272223ec.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-34-125-198.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2dea8f8b5e44e8fd6ef34a7ee9507f,SystemUUID:ec2dea8f-8b5e-44e8-fd6e-f34a7ee9507f,BootID:d41641dd-09ee-4029-b1ae-fb0f0a4ef62f,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d kubernetes.io/csi/ebs.csi.aws.com^vol-054fcfb0606bc608a],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-033820b09ac08fb6d,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/ebs.csi.aws.com^vol-054fcfb0606bc608a,DevicePath:,},},Config:nil,},} Mar 27 23:55:14.541: INFO: Logging kubelet events for node i-05ec0d9f3272223ec Mar 27 23:55:14.727: INFO: Logging pods the kubelet thinks is on node i-05ec0d9f3272223ec Mar 27 23:55:14.913: INFO: probe-test-044ed117-c53a-4e96-8fd7-3e37a6cefce1 started at 2023-03-27 23:55:06 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:14.913: INFO: Container probe-test-044ed117-c53a-4e96-8fd7-3e37a6cefce1 ready: true, restart count 0 Mar 27 23:55:14.913: INFO: ss-1 started at 2023-03-27 23:54:56 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:14.913: INFO: Container webserver ready: true, restart count 0 Mar 27 23:55:14.913: INFO: httpd started at 2023-03-27 23:54:23 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:14.913: INFO: Container httpd ready: true, restart count 0 Mar 27 23:55:14.913: INFO: up-down-2-jsqpk started at 2023-03-27 23:54:19 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:14.913: INFO: Container up-down-2 ready: true, restart count 0 Mar 27 23:55:14.913: INFO: netserver-1 started at 2023-03-27 23:54:44 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:14.913: INFO: Container webserver ready: true, restart count 0 Mar 27 23:55:14.913: INFO: verify-service-up-host-exec-pod started at 2023-03-27 23:55:05 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:14.913: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:14.913: INFO: kube-proxy-i-05ec0d9f3272223ec started at 2023-03-27 23:25:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:14.913: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:55:14.913: INFO: coredns-559769c974-m6hh6 started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:14.913: INFO: Container coredns ready: true, restart count 0 Mar 27 23:55:14.913: INFO: inline-volume-tester-kwwd2 started at 2023-03-27 23:43:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:14.913: INFO: Container csi-volume-tester ready: false, restart count 0 Mar 27 23:55:14.913: INFO: pod-ephm-test-secret-jtp6 started at 2023-03-27 23:53:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:14.913: INFO: Container test-container-subpath-secret-jtp6 ready: false, restart count 0 Mar 27 23:55:14.913: INFO: ebs-csi-node-gjw52 started at 2023-03-27 23:25:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:55:14.913: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:55:14.913: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:55:14.913: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:55:14.913: INFO: up-down-3-kzgf9 started at 2023-03-27 23:55:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:14.913: INFO: Container up-down-3 ready: true, restart count 0 Mar 27 23:55:14.913: INFO: test-container-pod started at 2023-03-27 23:55:08 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:14.913: INFO: Container webserver ready: true, restart count 0 Mar 27 23:55:14.913: INFO: pod1 started at 2023-03-27 23:54:57 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:14.913: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:14.913: INFO: calico-node-5lq5m started at 2023-03-27 23:25:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:55:14.913: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:55:14.913: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:55:14.913: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:55:14.913: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:55:15.803: INFO: Latency metrics for node i-05ec0d9f3272223ec Mar 27 23:55:15.803: INFO: Logging node info for node i-0b266cca2f40af1c1 Mar 27 23:55:15.987: INFO: Node Info: &Node{ObjectMeta:{i-0b266cca2f40af1c1 5f957448-9afc-4966-b9d6-82332b7a159d 57752 0 2023-03-27 23:29:17 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0b266cca2f40af1c1 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0b266cca2f40af1c1 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-7886":"i-0b266cca2f40af1c1","csi-mock-csi-mock-volumes-expansion-2991":"i-0b266cca2f40af1c1","ebs.csi.aws.com":"i-0b266cca2f40af1c1"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.38.36/19 projectcalico.org/IPv4IPIPTunnelAddr:100.116.128.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:29:17 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:30:03 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.8.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:30:13 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2023-03-27 23:54:58 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status} {kubelet Update v1 2023-03-27 23:55:14 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.8.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0b266cca2f40af1c1,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.8.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:30:13 +0000 UTC,LastTransitionTime:2023-03-27 23:30:13 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:58 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:58 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:54:58 +0000 UTC,LastTransitionTime:2023-03-27 23:29:17 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:54:58 +0000 UTC,LastTransitionTime:2023-03-27 23:30:03 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.38.36,},NodeAddress{Type:ExternalIP,Address:13.209.22.95,},NodeAddress{Type:InternalDNS,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0b266cca2f40af1c1.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-13-209-22-95.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec247601b7b315369bfb310559a7772d,SystemUUID:ec247601-b7b3-1536-9bfb-310559a7772d,BootID:487b59c3-9394-4a90-a6ec-b20c4819fe9e,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/sample-apiserver@sha256:8d70890151aa5d096f331cb9da1b9cd5be0412b7363fe67b5c3befdcaa2a28d0 registry.k8s.io/e2e-test-images/sample-apiserver:1.17.7],SizeBytes:25667066,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/regression-issue-74839@sha256:b4f1d8d61bdad84bd50442d161d5460e4019d53e989b64220fdbc62fc87d76bf registry.k8s.io/e2e-test-images/regression-issue-74839:1.2],SizeBytes:18651485,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/csi-hostpath-ephemeral-7886^c04b2cb6-ccfa-11ed-a1b1-fa34c4e6c5a5 kubernetes.io/csi/csi-hostpath-ephemeral-7886^c4676d58-ccfa-11ed-a1b1-fa34c4e6c5a5 kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-2991^9ab3bf37-ccfa-11ed-a28d-ead8e583275f],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-expansion-2991^9ab3bf37-ccfa-11ed-a28d-ead8e583275f,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7886^c04b2cb6-ccfa-11ed-a1b1-fa34c4e6c5a5,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/csi-hostpath-ephemeral-7886^c4676d58-ccfa-11ed-a1b1-fa34c4e6c5a5,DevicePath:,},},Config:nil,},} Mar 27 23:55:15.988: INFO: Logging kubelet events for node i-0b266cca2f40af1c1 Mar 27 23:55:16.175: INFO: Logging pods the kubelet thinks is on node i-0b266cca2f40af1c1 Mar 27 23:55:16.359: INFO: pfpod started at 2023-03-27 23:52:51 +0000 UTC (0+2 container statuses recorded) Mar 27 23:55:16.359: INFO: Container portforwardtester ready: false, restart count 0 Mar 27 23:55:16.359: INFO: Container readiness ready: false, restart count 0 Mar 27 23:55:16.359: INFO: up-down-2-w7vfg started at 2023-03-27 23:54:19 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:16.359: INFO: Container up-down-2 ready: true, restart count 0 Mar 27 23:55:16.359: INFO: kube-proxy-i-0b266cca2f40af1c1 started at 2023-03-27 23:29:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:16.359: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:55:16.359: INFO: calico-node-nmfrn started at 2023-03-27 23:29:18 +0000 UTC (3+1 container statuses recorded) Mar 27 23:55:16.359: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:55:16.359: INFO: Init container install-cni ready: true, restart count 1 Mar 27 23:55:16.359: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:55:16.359: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:55:16.359: INFO: inline-volume-tester-8gmqj started at 2023-03-27 23:54:46 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:16.359: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 27 23:55:16.359: INFO: csi-mockplugin-0 started at 2023-03-27 23:53:39 +0000 UTC (0+3 container statuses recorded) Mar 27 23:55:16.359: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:55:16.359: INFO: Container driver-registrar ready: true, restart count 0 Mar 27 23:55:16.359: INFO: Container mock ready: true, restart count 0 Mar 27 23:55:16.359: INFO: csi-mockplugin-resizer-0 started at 2023-03-27 23:53:40 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:16.359: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:55:16.359: INFO: csi-mockplugin-attacher-0 started at 2023-03-27 23:53:39 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:16.359: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:55:16.359: INFO: pvc-volume-tester-kjswl started at 2023-03-27 23:53:43 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:16.359: INFO: Container volume-tester ready: false, restart count 0 Mar 27 23:55:16.359: INFO: csi-hostpathplugin-0 started at 2023-03-27 23:54:43 +0000 UTC (0+7 container statuses recorded) Mar 27 23:55:16.359: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:55:16.359: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:55:16.359: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:55:16.359: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 27 23:55:16.359: INFO: Container hostpath ready: true, restart count 0 Mar 27 23:55:16.359: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:55:16.359: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:55:16.359: INFO: ebs-csi-node-ztjm9 started at 2023-03-27 23:29:18 +0000 UTC (0+3 container statuses recorded) Mar 27 23:55:16.359: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:55:16.359: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:55:16.359: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:55:16.359: INFO: inline-volume-tester2-4j4zs started at 2023-03-27 23:54:53 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:16.359: INFO: Container csi-volume-tester ready: true, restart count 0 Mar 27 23:55:16.359: INFO: netserver-2 started at 2023-03-27 23:54:44 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:16.359: INFO: Container webserver ready: true, restart count 0 Mar 27 23:55:17.010: INFO: Latency metrics for node i-0b266cca2f40af1c1 Mar 27 23:55:17.010: INFO: Logging node info for node i-0d2b1184f62de758a Mar 27 23:55:17.188: INFO: Node Info: &Node{ObjectMeta:{i-0d2b1184f62de758a 2e42ccb1-9580-4126-b5d6-05a5f98403c5 45677 0 2023-03-27 23:17:33 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kops.k8s.io/kops-controller-pki: kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0d2b1184f62de758a kubernetes.io/os:linux node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0d2b1184f62de758a"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.43.6/19 projectcalico.org/IPv4IPIPTunnelAddr:100.120.150.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-27 23:17:33 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {protokube Update v1 2023-03-27 23:17:34 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/kops-controller-pki":{},"f:node-role.kubernetes.io/control-plane":{},"f:node.kubernetes.io/exclude-from-external-load-balancers":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:18:48 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.5.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:18:49 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:taints":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:18:57 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:50:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.5.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0d2b1184f62de758a,Unschedulable:false,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/control-plane,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[100.96.5.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3892310016 0} {<nil>} 3801084Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3787452416 0} {<nil>} 3698684Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:18:49 +0000 UTC,LastTransitionTime:2023-03-27 23:18:49 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:17:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:50:52 +0000 UTC,LastTransitionTime:2023-03-27 23:18:45 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.43.6,},NodeAddress{Type:ExternalIP,Address:3.35.11.126,},NodeAddress{Type:InternalDNS,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0d2b1184f62de758a.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-3-35-11-126.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2781baf4898d15db3c853bf5b6d100,SystemUUID:ec2781ba-f489-8d15-db3c-853bf5b6d100,BootID:5e905df0-494e-4050-b6d7-400763c3dd66,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/etcd@sha256:dd75ec974b0a2a6f6bb47001ba09207976e625db898d1b16735528c009cb171c registry.k8s.io/etcd:3.5.6-0],SizeBytes:102542580,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:6f72b851544986cb0921b53ea655ec04c36131248f16d4ad110cb3ca0c369dc1 registry.k8s.io/etcd:3.5.4-0],SizeBytes:102157811,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5 registry.k8s.io/etcd:3.5.3-0],SizeBytes:102143581,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646 registry.k8s.io/etcd:3.4.3-0],SizeBytes:100947667,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:9ce33ba33d8e738a5b85ed50b5080ac746deceed4a7496c550927a7a19ca3b6d registry.k8s.io/etcd:3.5.0-0],SizeBytes:99868722,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:64b9ea357325d5db9f8a723dcf503b5a449177b17ac87d69481e126bb724c263 registry.k8s.io/etcd:3.5.1-0],SizeBytes:98888614,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:4ad90a11b55313b182afc186b9876c8e891531b8db4c9bf1541953021618d0e2 registry.k8s.io/etcd:3.4.13-0],SizeBytes:86742272,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:0d2ab6b15b4fe0b903b9e81192d942fb39c29be61fe341cd16db5d0165b0435c registry.k8s.io/etcd:3.3.17-0],SizeBytes:85524408,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:17da501f5d2a675be46040422a27b7cc21b8a43895ac998b171db1c346f361f7 registry.k8s.io/etcd:3.3.10-0],SizeBytes:76162358,},ContainerImage{Names:[registry.k8s.io/etcdadm/etcd-manager-slim@sha256:8727adc059334e0e3329dbccce116a734da4a090da5e86c8cddccaf333caa666 registry.k8s.io/etcdadm/etcd-manager-slim:v3.0.20230201],SizeBytes:63926165,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20 registry.k8s.io/etcd:3.2.24-1],SizeBytes:63423287,},ContainerImage{Names:[registry.k8s.io/kops/kops-controller:1.27.0-alpha.1],SizeBytes:45208814,},ContainerImage{Names:[registry.k8s.io/kops/dns-controller:1.27.0-alpha.1],SizeBytes:43629012,},ContainerImage{Names:[registry.k8s.io/kube-apiserver@sha256:49934a29c559e79ed19fc38b4697b18d8fe174b17bf56192588575f980c93956 registry.k8s.io/kube-apiserver:v1.28.0-alpha.0],SizeBytes:33212506,},ContainerImage{Names:[docker.io/calico/kube-controllers@sha256:2b6acd7f677f76ffe12ecf3ea7df92eb9b1bdb07336d1ac2a54c7631fb753f7e docker.io/calico/kube-controllers:v3.24.5],SizeBytes:31137169,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager@sha256:974748384634982c73527fdb3c9ec0ec476bc0d52b3dd1d611a380b7f99af1f3 registry.k8s.io/kube-controller-manager:v1.28.0-alpha.0],SizeBytes:30820280,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:122bfb8c1edabb3c0edd63f06523e6940d958d19b3957dc7b1d6f81e9f1f6119 registry.k8s.io/sig-storage/csi-provisioner:v3.1.0],SizeBytes:23345856,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:9ebbf9f023e7b41ccee3d52afe39a89e3ddacdbb69269d583abfc25847cfd9e4 registry.k8s.io/sig-storage/csi-resizer:v1.4.0],SizeBytes:22381475,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:8b9c313c05f54fb04f8d430896f5f5904b6cb157df261501b29adc04d2b2dc7b registry.k8s.io/sig-storage/csi-attacher:v3.4.0],SizeBytes:22085298,},ContainerImage{Names:[registry.k8s.io/provider-aws/cloud-controller-manager@sha256:fdeb61e3e42ecd9cca868d550ebdb88dd6341d9e91fcfa9a37e227dab2ad22cb registry.k8s.io/provider-aws/cloud-controller-manager:v1.26.0],SizeBytes:20154862,},ContainerImage{Names:[registry.k8s.io/kube-scheduler@sha256:cbf4007abb076188f16b975bc2b74452603cc7b34c60a29d92f14c38ad02dc11 registry.k8s.io/kube-scheduler:v1.28.0-alpha.0],SizeBytes:18078674,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/kops/kube-apiserver-healthcheck:1.27.0-alpha.1],SizeBytes:5074727,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:55:17.188: INFO: Logging kubelet events for node i-0d2b1184f62de758a Mar 27 23:55:17.379: INFO: Logging pods the kubelet thinks is on node i-0d2b1184f62de758a Mar 27 23:55:17.565: INFO: kube-scheduler-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:17.565: INFO: Container kube-scheduler ready: true, restart count 0 Mar 27 23:55:17.565: INFO: dns-controller-69987775c6-rzzjj started at 2023-03-27 23:18:25 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:17.565: INFO: Container dns-controller ready: true, restart count 0 Mar 27 23:55:17.565: INFO: calico-node-scbv2 started at 2023-03-27 23:18:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:55:17.565: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:55:17.565: INFO: ebs-csi-node-nvqcx started at 2023-03-27 23:18:29 +0000 UTC (0+3 container statuses recorded) Mar 27 23:55:17.565: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:55:17.565: INFO: kops-controller-w4b7t started at 2023-03-27 23:18:29 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:17.565: INFO: Container kops-controller ready: true, restart count 0 Mar 27 23:55:17.565: INFO: etcd-manager-events-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:55:17.565: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:55:17.565: INFO: etcd-manager-main-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (11+1 container statuses recorded) Mar 27 23:55:17.565: INFO: Init container init-etcd-3-2-24 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-3-10 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-3-17 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-4-13 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-4-3 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-5-0 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-5-1 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-5-3 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-5-4 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-5-6 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Init container init-etcd-3-5-7 ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Container etcd-manager ready: true, restart count 0 Mar 27 23:55:17.565: INFO: kube-apiserver-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+2 container statuses recorded) Mar 27 23:55:17.565: INFO: Container healthcheck ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Container kube-apiserver ready: true, restart count 3 Mar 27 23:55:17.565: INFO: kube-controller-manager-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:17.565: INFO: Container kube-controller-manager ready: true, restart count 4 Mar 27 23:55:17.565: INFO: kube-proxy-i-0d2b1184f62de758a started at 2023-03-27 23:15:37 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:17.565: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:55:17.565: INFO: aws-cloud-controller-manager-776ld started at 2023-03-27 23:18:28 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:17.565: INFO: Container aws-cloud-controller-manager ready: true, restart count 0 Mar 27 23:55:17.565: INFO: calico-kube-controllers-55f488656-vj6gh started at 2023-03-27 23:18:57 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:17.565: INFO: Container calico-kube-controllers ready: true, restart count 0 Mar 27 23:55:17.565: INFO: ebs-csi-controller-79f984545f-42vd6 started at 2023-03-27 23:19:48 +0000 UTC (0+5 container statuses recorded) Mar 27 23:55:17.565: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:55:17.565: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:55:18.267: INFO: Latency metrics for node i-0d2b1184f62de758a Mar 27 23:55:18.267: INFO: Logging node info for node i-0e1f2cd7cf96d3020 Mar 27 23:55:18.449: INFO: Node Info: &Node{ObjectMeta:{i-0e1f2cd7cf96d3020 efcf1e66-db0e-4256-a849-b9c377eef04e 57851 0 2023-03-27 23:22:27 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-0e1f2cd7cf96d3020 kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-0e1f2cd7cf96d3020 topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-0e1f2cd7cf96d3020"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.48.120/19 projectcalico.org/IPv4IPIPTunnelAddr:100.107.86.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:22:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:22:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.6.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:22:53 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:55:16 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.6.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-0e1f2cd7cf96d3020,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.6.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:22:52 +0000 UTC,LastTransitionTime:2023-03-27 23:22:52 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:55:16 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:55:16 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:55:16 +0000 UTC,LastTransitionTime:2023-03-27 23:22:27 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:55:16 +0000 UTC,LastTransitionTime:2023-03-27 23:22:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.48.120,},NodeAddress{Type:ExternalIP,Address:15.164.224.157,},NodeAddress{Type:InternalDNS,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-0e1f2cd7cf96d3020.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-15-164-224-157.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec29484c87c97fb3d8a52788d5971660,SystemUUID:ec29484c-87c9-7fb3-d8a5-2788d5971660,BootID:b73c540f-bd49-470a-a7b3-08c31cc8c2fd,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:68d396900aeaa072c1f27289485fdac29834045a6f3ffe369bf389d830ef572d registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6],SizeBytes:20293261,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf registry.k8s.io/e2e-test-images/busybox:1.29-2],SizeBytes:732424,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[kubernetes.io/csi/ebs.csi.aws.com^vol-002c1b95622403bc5],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:55:18.449: INFO: Logging kubelet events for node i-0e1f2cd7cf96d3020 Mar 27 23:55:18.633: INFO: Logging pods the kubelet thinks is on node i-0e1f2cd7cf96d3020 Mar 27 23:55:18.825: INFO: coredns-autoscaler-7cb5c5b969-2k8xf started at 2023-03-27 23:26:22 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:18.825: INFO: Container autoscaler ready: true, restart count 0 Mar 27 23:55:18.825: INFO: ss-2 started at 2023-03-27 23:55:10 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:18.825: INFO: Container webserver ready: false, restart count 0 Mar 27 23:55:18.825: INFO: hostexec-i-0e1f2cd7cf96d3020-h74tq started at 2023-03-27 23:55:04 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:18.825: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:18.825: INFO: pod-subpath-test-preprovisionedpv-sk29 started at 2023-03-27 23:55:16 +0000 UTC (2+2 container statuses recorded) Mar 27 23:55:18.825: INFO: Init container init-volume-preprovisionedpv-sk29 ready: true, restart count 0 Mar 27 23:55:18.825: INFO: Init container test-init-subpath-preprovisionedpv-sk29 ready: false, restart count 0 Mar 27 23:55:18.825: INFO: Container test-container-subpath-preprovisionedpv-sk29 ready: false, restart count 0 Mar 27 23:55:18.825: INFO: Container test-container-volume-preprovisionedpv-sk29 ready: false, restart count 0 Mar 27 23:55:18.825: INFO: funny-ip-dqzj4 started at 2023-03-27 23:55:17 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:18.825: INFO: Container funny-ip ready: false, restart count 0 Mar 27 23:55:18.825: INFO: pod-csi-inline-volumes started at 2023-03-27 23:54:59 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:18.825: INFO: Container pod-csi-inline-volumes ready: false, restart count 0 Mar 27 23:55:18.825: INFO: up-down-3-r5hn8 started at 2023-03-27 23:55:02 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:18.825: INFO: Container up-down-3 ready: true, restart count 0 Mar 27 23:55:18.825: INFO: coredns-559769c974-scn7m started at 2023-03-27 23:30:58 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:18.825: INFO: Container coredns ready: true, restart count 0 Mar 27 23:55:18.825: INFO: verify-service-up-exec-pod-xcgd6 started at 2023-03-27 23:55:18 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:18.825: INFO: Container agnhost-container ready: false, restart count 0 Mar 27 23:55:18.825: INFO: execpodqt86v started at 2023-03-27 23:55:00 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:18.825: INFO: Container agnhost-container ready: true, restart count 0 Mar 27 23:55:18.825: INFO: ebs-csi-node-vcn4h started at 2023-03-27 23:22:28 +0000 UTC (0+3 container statuses recorded) Mar 27 23:55:18.825: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:55:18.825: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:55:18.825: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:55:18.825: INFO: busybox-user-0-7b4fa815-8598-4cc0-b651-e0ca8b753e70 started at 2023-03-27 23:55:13 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:18.825: INFO: Container busybox-user-0-7b4fa815-8598-4cc0-b651-e0ca8b753e70 ready: false, restart count 0 Mar 27 23:55:18.825: INFO: kube-proxy-i-0e1f2cd7cf96d3020 started at 2023-03-27 23:22:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:55:18.825: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:55:18.825: INFO: calico-node-9fzr7 started at 2023-03-27 23:22:28 +0000 UTC (3+1 container statuses recorded) Mar 27 23:55:18.825: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:55:18.825: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:55:18.825: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:55:18.825: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:55:19.629: INFO: Latency metrics for node i-0e1f2cd7cf96d3020 END STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:55:19.629 (7.212s) < Exit [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - dump namespaces | framework.go:209 @ 03/27/23 23:55:19.629 (7.212s) > Enter [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - tear down framework | framework.go:206 @ 03/27/23 23:55:19.629 STEP: Destroying namespace "ephemeral-7265" for this suite. - test/e2e/framework/framework.go:351 @ 03/27/23 23:55:19.629 < Exit [DeferCleanup (Each)] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral - tear down framework | framework.go:206 @ 03/27/23 23:55:19.808 (178ms) > Enter [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:55:19.808 < Exit [ReportAfterEach] TOP-LEVEL - test/e2e/e2e_test.go:144 @ 03/27/23 23:55:19.808 (0s)
Find to mentions in log files | View test history on testgrid
go run hack/e2e.go -v --test --test_args='--ginkgo.focus=Kubernetes\se2e\ssuite\s\[It\]\s\[sig\-auth\]\sServiceAccounts\sServiceAccountIssuerDiscovery\sshould\ssupport\sOIDC\sdiscovery\sof\sservice\saccount\sissuer\s\[Conformance\]$'
[FAILED] Unexpected error getting pod logs: timed out waiting for the condition In [It] at: test/e2e/auth/service_accounts.go:630 @ 03/27/23 23:49:38.419
> Enter [BeforeEach] [sig-auth] ServiceAccounts - set up framework | framework.go:191 @ 03/27/23 23:47:31.737 STEP: Creating a kubernetes client - test/e2e/framework/framework.go:211 @ 03/27/23 23:47:31.737 Mar 27 23:47:31.737: INFO: >>> kubeConfig: /root/.kube/config STEP: Building a namespace api object, basename svcaccounts - test/e2e/framework/framework.go:250 @ 03/27/23 23:47:31.738 STEP: Waiting for a default service account to be provisioned in namespace - test/e2e/framework/framework.go:259 @ 03/27/23 23:47:32.273 STEP: Waiting for kube-root-ca.crt to be provisioned in namespace - test/e2e/framework/framework.go:262 @ 03/27/23 23:47:32.626 < Exit [BeforeEach] [sig-auth] ServiceAccounts - set up framework | framework.go:191 @ 03/27/23 23:47:32.98 (1.242s) > Enter [BeforeEach] [sig-auth] ServiceAccounts - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:47:32.98 < Exit [BeforeEach] [sig-auth] ServiceAccounts - test/e2e/framework/metrics/init/init.go:33 @ 03/27/23 23:47:32.98 (0s) > Enter [It] ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance] - test/e2e/auth/service_accounts.go:529 @ 03/27/23 23:47:32.98 Mar 27 23:47:33.339: INFO: created pod STEP: Saw pod success - test/e2e/framework/pod/wait.go:409 @ 03/27/23 23:47:37.884 Mar 27 23:48:07.885: INFO: polling logs Mar 27 23:48:08.062: INFO: Error pulling logs: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-3730/pods/oidc-discovery-validator/log?container=oidc-discovery-validator&previous=false": stream error: stream ID 1799; INTERNAL_ERROR; received from peer Mar 27 23:48:37.885: INFO: polling logs Mar 27 23:48:38.066: INFO: Error pulling logs: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-3730/pods/oidc-discovery-validator/log?container=oidc-discovery-validator&previous=false": stream error: stream ID 1801; INTERNAL_ERROR; received from peer Mar 27 23:49:07.885: INFO: polling logs Mar 27 23:49:08.065: INFO: Error pulling logs: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-3730/pods/oidc-discovery-validator/log?container=oidc-discovery-validator&previous=false": stream error: stream ID 1803; INTERNAL_ERROR; received from peer Mar 27 23:49:37.885: INFO: polling logs Mar 27 23:49:38.062: INFO: Error pulling logs: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-3730/pods/oidc-discovery-validator/log?container=oidc-discovery-validator&previous=false": stream error: stream ID 1805; INTERNAL_ERROR; received from peer Mar 27 23:49:38.062: INFO: polling logs Mar 27 23:49:38.239: INFO: Error pulling logs: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-3730/pods/oidc-discovery-validator/log?container=oidc-discovery-validator&previous=false": stream error: stream ID 1807; INTERNAL_ERROR; received from peer [FAILED] Unexpected error getting pod logs: timed out waiting for the condition In [It] at: test/e2e/auth/service_accounts.go:630 @ 03/27/23 23:49:38.419 < Exit [It] ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance] - test/e2e/auth/service_accounts.go:529 @ 03/27/23 23:49:38.419 (2m5.44s) > Enter [AfterEach] [sig-auth] ServiceAccounts - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:49:38.419 Mar 27 23:49:38.419: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready < Exit [AfterEach] [sig-auth] ServiceAccounts - test/e2e/framework/node/init/init.go:33 @ 03/27/23 23:49:38.599 (179ms) > Enter [DeferCleanup (Each)] [sig-auth] ServiceAccounts - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:49:38.599 < Exit [DeferCleanup (Each)] [sig-auth] ServiceAccounts - test/e2e/framework/metrics/init/init.go:35 @ 03/27/23 23:49:38.599 (0s) > Enter [DeferCleanup (Each)] [sig-auth] ServiceAccounts - dump namespaces | framework.go:209 @ 03/27/23 23:49:38.599 STEP: dump namespace information after failure - test/e2e/framework/framework.go:288 @ 03/27/23 23:49:38.599 STEP: Collecting events from namespace "svcaccounts-3730". - test/e2e/framework/debug/dump.go:42 @ 03/27/23 23:49:38.599 STEP: Found 4 events. - test/e2e/framework/debug/dump.go:46 @ 03/27/23 23:49:38.776 Mar 27 23:49:38.776: INFO: At 2023-03-27 23:47:33 +0000 UTC - event for oidc-discovery-validator: {default-scheduler } Scheduled: Successfully assigned svcaccounts-3730/oidc-discovery-validator to i-022003b3a45876d5d Mar 27 23:49:38.776: INFO: At 2023-03-27 23:47:34 +0000 UTC - event for oidc-discovery-validator: {kubelet i-022003b3a45876d5d} Pulled: Container image "registry.k8s.io/e2e-test-images/agnhost:2.43" already present on machine Mar 27 23:49:38.776: INFO: At 2023-03-27 23:47:34 +0000 UTC - event for oidc-discovery-validator: {kubelet i-022003b3a45876d5d} Created: Created container oidc-discovery-validator Mar 27 23:49:38.776: INFO: At 2023-03-27 23:47:34 +0000 UTC - event for oidc-discovery-validator: {kubelet i-022003b3a45876d5d} Started: Started container oidc-discovery-validator Mar 27 23:49:38.954: INFO: POD NODE PHASE GRACE CONDITIONS Mar 27 23:49:38.954: INFO: oidc-discovery-validator i-022003b3a45876d5d Succeeded [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2023-03-27 23:47:33 +0000 UTC PodCompleted } {Ready False 0001-01-01 00:00:00 +0000 UTC 2023-03-27 23:47:33 +0000 UTC PodCompleted } {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2023-03-27 23:47:33 +0000 UTC PodCompleted } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2023-03-27 23:47:33 +0000 UTC }] Mar 27 23:49:38.954: INFO: Mar 27 23:49:39.131: INFO: Unable to fetch svcaccounts-3730/oidc-discovery-validator/oidc-discovery-validator logs: Get "https://api.e2e-3a5daf4e9a-96255.test-cncf-aws.k8s.io/api/v1/namespaces/svcaccounts-3730/pods/oidc-discovery-validator/log?container=oidc-discovery-validator&previous=false&tailLines=42": stream error: stream ID 1817; INTERNAL_ERROR; received from peer Mar 27 23:49:39.310: INFO: Logging node info for node i-022003b3a45876d5d Mar 27 23:49:39.497: INFO: Node Info: &Node{ObjectMeta:{i-022003b3a45876d5d 20348b58-b610-41a0-accb-ca195a540336 42684 0 2023-03-27 23:33:26 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:t3.medium beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:ap-northeast-2 failure-domain.beta.kubernetes.io/zone:ap-northeast-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:i-022003b3a45876d5d kubernetes.io/os:linux node-role.kubernetes.io/node: node.kubernetes.io/instance-type:t3.medium topology.ebs.csi.aws.com/zone:ap-northeast-2c topology.hostpath.csi/node:i-022003b3a45876d5d topology.kubernetes.io/region:ap-northeast-2 topology.kubernetes.io/zone:ap-northeast-2c] map[csi.volume.kubernetes.io/nodeid:{"csi-hostpath-ephemeral-1547":"i-022003b3a45876d5d","ebs.csi.aws.com":"i-022003b3a45876d5d"} node.alpha.kubernetes.io/ttl:0 projectcalico.org/IPv4Address:172.20.49.118/19 projectcalico.org/IPv4IPIPTunnelAddr:100.115.192.64 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {aws-cloud-controller-manager Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:status":{"f:addresses":{"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{"f:address":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}}}}} status} {kops-controller Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:node-role.kubernetes.io/node":{}}}} } {kubelet Update v1 2023-03-27 23:33:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{}}}} } {kube-controller-manager Update v1 2023-03-27 23:33:43 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.9.0/24\"":{}}}} } {calico-node Update v1 2023-03-27 23:33:52 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:projectcalico.org/IPv4Address":{},"f:projectcalico.org/IPv4IPIPTunnelAddr":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kubelet Update v1 2023-03-27 23:49:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.ebs.csi.aws.com/zone":{},"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:100.96.9.0/24,DoNotUseExternalID:,ProviderID:aws:///ap-northeast-2c/i-022003b3a45876d5d,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.9.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{49753808896 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{4051681280 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{44778427933 0} {<nil>} 44778427933 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3946823680 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-27 23:33:51 +0000 UTC,LastTransitionTime:2023-03-27 23:33:51 +0000 UTC,Reason:CalicoIsUp,Message:Calico is running on this node,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-27 23:49:25 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-27 23:49:25 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-27 23:49:25 +0000 UTC,LastTransitionTime:2023-03-27 23:33:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-27 23:49:25 +0000 UTC,LastTransitionTime:2023-03-27 23:33:43 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.49.118,},NodeAddress{Type:ExternalIP,Address:54.180.103.89,},NodeAddress{Type:InternalDNS,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:Hostname,Address:i-022003b3a45876d5d.ap-northeast-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-54-180-103-89.ap-northeast-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2ccb51411bbbb92946da93b9a452fe,SystemUUID:ec2ccb51-411b-bbb9-2946-da93b9a452fe,BootID:841d91a9-bfa5-4e0f-9279-7c83efeef8ca,KernelVersion:5.15.0-1031-aws,OSImage:Ubuntu 20.04.5 LTS,ContainerRuntimeVersion:containerd://1.6.18,KubeletVersion:v1.28.0-alpha.0,KubeProxyVersion:v1.28.0-alpha.0,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/e2e-test-images/jessie-dnsutils@sha256:24aaf2626d6b27864c29de2097e8bbb840b3a414271bf7c8995e431e47d8408e registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7],SizeBytes:112030336,},ContainerImage{Names:[docker.io/calico/cni@sha256:e282ea2914c806b5de2976330a17cfb5e6dcef47147bceb1432ca5c75fd46f50 docker.io/calico/cni:v3.24.5],SizeBytes:87458443,},ContainerImage{Names:[docker.io/calico/node@sha256:5972ad2bcbdc668564d3e26960c9c513b2d7b05581c704747cf7c62ef3a405a6 docker.io/calico/node:v3.24.5],SizeBytes:81584866,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/agnhost@sha256:16bbf38c463a4223d8cfe4da12bc61010b082a79b4bb003e2d3ba3ece5dd5f9e registry.k8s.io/e2e-test-images/agnhost:2.43],SizeBytes:51706353,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nautilus@sha256:80ba6c8c44f9623f06e868a1aa66026c8ec438ad814f9ec95e9333b415fe3550 registry.k8s.io/e2e-test-images/nautilus:1.7],SizeBytes:49641698,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:3fe7acf013d1264ffded116b80a73dc129a449b0fccdb8d21af8279f2233f36e registry.k8s.io/e2e-test-images/httpd:2.4.39-4],SizeBytes:41901587,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/httpd@sha256:148b022f5c5da426fc2f3c14b5c0867e58ef05961510c84749ac1fddcb0fef22 registry.k8s.io/e2e-test-images/httpd:2.4.38-4],SizeBytes:40764257,},ContainerImage{Names:[registry.k8s.io/provider-aws/aws-ebs-csi-driver@sha256:f0c5de192d832e7c1daa6580d4a62e8fa6fc8eabc0917ae4cb7ed4d15e95b59e registry.k8s.io/provider-aws/aws-ebs-csi-driver:v1.14.1],SizeBytes:29725845,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-provisioner@sha256:e468dddcd275163a042ab297b2d8c2aca50d5e148d2d22f3b6ba119e2f31fa79 registry.k8s.io/sig-storage/csi-provisioner:v3.4.0],SizeBytes:27427836,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-resizer@sha256:425d8f1b769398127767b06ed97ce62578a3179bcb99809ce93a1649e025ffe7 registry.k8s.io/sig-storage/csi-resizer:v1.6.0],SizeBytes:24148884,},ContainerImage{Names:[registry.k8s.io/kube-proxy@sha256:8a03b3147cd7be8e8235ae1cd70cf69aca9ce03ba1cfaffea320540bb5afa496 registry.k8s.io/kube-proxy:v1.28.0-alpha.0],SizeBytes:23896850,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-snapshotter@sha256:291334908ddf71a4661fd7f6d9d97274de8a5378a2b6fdfeb2ce73414a34f82f registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0],SizeBytes:23881995,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-attacher@sha256:9a685020911e2725ad019dbce6e4a5ab93d51e3d4557f115e64343345e05781b registry.k8s.io/sig-storage/csi-attacher:v4.0.0],SizeBytes:23847201,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:92257881c1d6493cf18299a24af42330f891166560047902b8d431fb66b01af5 registry.k8s.io/sig-storage/hostpathplugin:v1.9.0],SizeBytes:18758628,},ContainerImage{Names:[registry.k8s.io/sig-storage/hostpathplugin@sha256:9b2d9abda017c28b12f84a344f57cd73fbdb6c2bd7dd5bdd5018246ad1093ba6 registry.k8s.io/sig-storage/hostpathplugin:v1.11.0],SizeBytes:18233005,},ContainerImage{Names:[registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1],SizeBytes:9133109,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:933940f13b3ea0abc62e656c1aa5c5b47c04b15d71250413a6b821bd0c58b94e registry.k8s.io/sig-storage/livenessprobe:v2.7.0],SizeBytes:8688564,},ContainerImage{Names:[registry.k8s.io/sig-storage/livenessprobe@sha256:406f59599991916d2942d8d02f076d957ed71b541ee19f09fc01723a6e6f5932 registry.k8s.io/sig-storage/livenessprobe:v2.6.0],SizeBytes:8240918,},ContainerImage{Names:[registry.k8s.io/build-image/distroless-iptables@sha256:13ac9b3c476d7290a4451d65d09d6016a2cb89836ffbfa4eae55b72731a22080 registry.k8s.io/build-image/distroless-iptables:v0.2.2],SizeBytes:7729580,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:db048754ae68ae337d8fa96494c96d2a1204c3320f5dcf7e8e71085adec85da6 registry.k8s.io/e2e-test-images/nginx:1.15-4],SizeBytes:7000082,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nginx@sha256:5c99cf6a02adda929b10321dbf4ecfa00d87be9ba4fb456006237d530ab4baa1 registry.k8s.io/e2e-test-images/nginx:1.14-4],SizeBytes:6978614,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac registry.k8s.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 registry.k8s.io/pause:3.9],SizeBytes:321520,},ContainerImage{Names:[registry.k8s.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db registry.k8s.io/pause:3.6],SizeBytes:301773,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} Mar 27 23:49:39.497: INFO: Logging kubelet events for node i-022003b3a45876d5d Mar 27 23:49:39.723: INFO: Logging pods the kubelet thinks is on node i-022003b3a45876d5d Mar 27 23:49:39.910: INFO: pod-subpath-test-downwardapi-4chp started at 2023-03-27 23:49:24 +0000 UTC (0+1 container statuses recorded) Mar 27 23:49:39.910: INFO: Container test-container-subpath-downwardapi-4chp ready: true, restart count 0 Mar 27 23:49:39.910: INFO: rs-h74jg started at 2023-03-27 23:49:34 +0000 UTC (0+1 container statuses recorded) Mar 27 23:49:39.910: INFO: Container donothing ready: true, restart count 0 Mar 27 23:49:39.910: INFO: kube-proxy-i-022003b3a45876d5d started at 2023-03-27 23:33:27 +0000 UTC (0+1 container statuses recorded) Mar 27 23:49:39.910: INFO: Container kube-proxy ready: true, restart count 0 Mar 27 23:49:39.910: INFO: ebs-csi-node-2knd4 started at 2023-03-27 23:33:27 +0000 UTC (0+3 container statuses recorded) Mar 27 23:49:39.910: INFO: Container ebs-plugin ready: true, restart count 0 Mar 27 23:49:39.910: INFO: Container liveness-probe ready: true, restart count 0 Mar 27 23:49:39.910: INFO: Container node-driver-registrar ready: true, restart count 0 Mar 27 23:49:39.910: INFO: oidc-discovery-validator started at 2023-03-27 23:47:33 +0000 UTC (0+1 container statuses recorded) Mar 27 23:49:39.910: INFO: Container oidc-discovery-validator ready: false, restart count 0 Mar 27 23:49:39.910: INFO: rs-kb7qq started at 2023-03-27 23:49:34 +0000 UTC (0+1 container statuses recorded) Mar 27 23:49:39.910: INFO: Container donothing ready: true, restart count 0 Mar 27 23:49:39.910: INFO: calico-node-2wjdb started at 2023-03-27 23:33:27 +0000 UTC (3+1 container statuses recorded) Mar 27 23:49:39.910: INFO: Init container upgrade-ipam ready: true, restart count 0 Mar 27 23:49:39.910: INFO: Init container install-cni ready: true, restart count 0 Mar 27 23:49:39.910: INFO: Init container mount-bpffs ready: true, restart count 0 Mar 27 23:49:39.910: INFO: Container calico-node ready: true, restart count 0 Mar 27 23:49:39.910: INFO: csi-hostpathplugin-0 started at 2023-03-27 23:47:55 +0000 UTC (0+7 container statuses recorded) Mar 27 23:49:39.910: INFO: Container csi-attacher ready: true, restart count 0 Mar 27 23:49:39.910: INFO: Container csi-provisioner ready: true, restart count 0 Mar 27 23:49:39.910: INFO: Container csi-resizer ready: true, restart count 0 Mar 27 23:49:39.910: INFO: Container csi-snapshotter ready: true, restart count 0 Mar 27 23:49:39.910: INFO: Container hostpath ready: true, restart count 0 Mar 27 23:49:39.910: INFO: Container liveness-probe